Table of Contents

• 1  LCS-SA Project: Vanderbijilpark, South Africa 2021.
  • 1.1  Objectives
  • 1.2  Libraries, Packages and General Functions
  • 1.3  Directory pathways
• 2  Import data
  • 2.1  Quality Control
    • 2.1.1  Apply report changes to data
    • 2.1.2  Remove Zero and Span & Site Vists
  • 2.2  Reference instruments
  • 2.3  MRC
    • 2.3.1  DF05 "05_Vaisala_S1830003_df5"
    • 2.3.2  TimeSeries Plots
    • 2.3.3  Scatterplots
    • 2.3.4  Accuracy Metrics
    • 2.3.5  DF06 "06_S500_5002-2D82-001_df6"
    • 2.3.6  TimeSeries Plots
    • 2.3.7  Scatterplots
    • 2.3.8  DF07 "07_S500_ECM-1906191-003_df7"
    • 2.3.9  Timeseries
    • 2.3.10  Scatterplot
  • 2.4  eroAfrica
    • 2.4.1  DF08 "08_PolludroneSmart_EA01P0001_df8"
    • 2.4.2  Timeseries
    • 2.4.3  Scatterplot
    • 2.4.4  DF17 "17_GM-5000_CM21035290_df17"
    • 2.4.5  Timeseries
    • 2.4.6  Scatterplot
    • 2.4.7  DF27 "27_PolludroneSmart_PM01P0007_df27"
    • 2.4.8  Timeseries
    • 2.4.9  Scatterplots
  • 2.5  Simplicity
    • 2.5.1  "09_SimplicityV1_CCSENV011_df9"
    • 2.5.2  Timeseries
    • 2.5.3  Scatterplots
    • 2.5.4  "10_SimplicityV1_CCSENV020_df10"
    • 2.5.5  Timeseries
    • 2.5.6  Scatterplots
    • 2.5.7  "11_SimplicityV2_IMTAQS0001_df11"
    • 2.5.8  Timeseries
    • 2.5.9  Scatterplots
    • 2.5.10  "12_SimplicityV2_IMTAQS0002_df12"
    • 2.5.11  Timeseries
    • 2.5.12  Scatterplots
    • 2.5.13  "13_SimplicityV2_IMTAQS0003_df13"
    • 2.5.14  Timeseries
    • 2.5.15  Scatterplots
    • 2.5.16  "14_SimplicityV2_IMTAQS0004_df14"
    • 2.5.17  Timeseries
    • 2.5.18  Scatterplots
  • 2.6  Envirocon
    • 2.6.1  DF15 "15_ICOMSMART_20149_df15"
    • 2.6.2  Timeseries
    • 2.6.3  Scatterplots
  • 2.7  Sensors.Africa
    • 2.7.1  DF18 "18_Plantower_108_df18"
    • 2.7.2  Timeseries
    • 2.7.3  Scatterplots
  • 2.8  University of L
    • 2.8.1  DF20 "20_Zephyr_642-SA_df20"
    • 2.8.2  Timeseries
    • 2.8.3  Scatterplots
    • 2.8.4  DF21 "21_Zephyr_729-SA_df21"
    • 2.8.5  Timeseries
    • 2.8.6  Scatterplots
  • 2.9  Skyside
    • 2.9.1  DF22 "22_Atmos_84CCA8B167D2_df22"
    • 2.9.2  Scatterplots
    • 2.9.3  DF23 "23_Atmos_98F4ABDCA328_df23"
    • 2.9.4  Scatterplots
  • 2.10  Sedulitas
    • 2.10.1  DF26 26_Zephyr_533_df26
  • 2.11  Scatterplots
  • 2.12  AfriqAir
  • 2.13  Data analysis
    • 2.13.1  Diurnal plots
    • 2.13.2  Data recovery
    • 2.13.3  Factors
    • 2.13.4  Reference air quality
    • 2.13.5  Bivariate plots

LCS-SA Project: Vanderbijilpark, South Africa 2021.

Objectives

Aim:

1. Environmental health in sub-Saharan-Africa - leveraging local and global air pollution data for epidemiological research (LEAP-Epi)

Objectives:

1. Determine the precision, accuracy, reliability, rigidity and usability within a South African setting.
2. Provide information which could be used to improve the low- and medium-cost sensors.

Specific research questions:

1. XXX
2. XXX

Libraries, Packages and General Functions

Back to the top

# Import the necessary packages.

#
import glob

#
from datetime import datetime
import pytz

# The matplotlip library is used to visualize data.
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import matplotlib.style
import matplotlib.dates as mdates
import matplotlib.cm as cm
from matplotlib.colors import BoundaryNorm
from matplotlib.ticker import MaxNLocator
import matplotlib.dates as mdates
from matplotlib.dates import DateFormatter
# Numpy is the numerical library. It is used by many of the ones before, but we import it for in case.
import numpy as np
from numpy.polynomial.polynomial import polyfit
# From Numpy we also import the nan 'missing-value' object that we use alot.
from numpy import nan

# The os library is used to list and access files and directories on the hard drive.
import os

# Used to manipulate data.
import pandas as pd

# Used to plot data.
import seaborn as sns
sns.set_theme(style="darkgrid")

# The pathlib library is used to manage access to disk. It helps us to keep things the same between Windows and Linux.
from pathlib import Path

# Used for statistical analysis such as boxplots and histograms.
from scipy import interpolate
from scipy.stats import linregress

# Used to create windroses or pollution plots.
from windrose import WindroseAxes

# For fancy matching of strings, we use re.
import re

# String libary for temporary file import.
from io import StringIO

#
import sklearn
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
# Create tooltips for interactive plots.
import mpld3

# For connecting to google sheet.
import gspread
from oauth2client.service_account import ServiceAccountCredentials

import ipywidgets as widgets
from ipywidgets import interact, interact_manual

timezonesa = pytz.timezone("Africa/Johannesburg")
timezoneutc = pytz.timezone("UTC")

# Configure the connection 
scope = ['https://spreadsheets.google.com/feeds']

# Give the path to the Service Account Credential json file 
# 1. Go to Google Development Console (https://console.developers.google.com/project)
# 2. Create new project
# 3. Go to “APIs & Services > Credentials” and choose “Create credentials > Service account key”.
# 4. Fill out the form
# 5. Click “Create key”
# 6. Select “JSON” and click “Create”

credentials = ServiceAccountCredentials.from_json_keyfile_name('key/crgprojects-b66f872bd816.json',
                                                               scope)
# Authorise your Notebook
gc = gspread.authorize(credentials)

# The sprad sheet ID, which can be taken from the link to the sheet
#https://docs.google.com/spreadsheets/d/1Fe1P2QEiCjl7KK6YEckzGLC_VqOuGtaChaDGbn2SOhE/edit?usp=sharing
#spreadsheet_key = '1pwJDzIl1O8gQSHD8SekW6442rrLRgLVTYA90dQb7chg'
spreadsheet_key = '1Fe1P2QEiCjl7KK6YEckzGLC_VqOuGtaChaDGbn2SOhE'

#Opening the worksheet by using Worksheet ID
def readSheet():
    workbook = gc.open_by_key(spreadsheet_key)#Selecting which sheet to pulling the data
    reports = workbook.worksheet('LCS-QC')#Pulling the data and transform it to the data frame
    values = reports.get_all_values()
    Reports = pd.DataFrame(values[1:], columns=values[0])
    #Reports=Reports[Reports['What is the site']=='Northam AQMS']
    return(reports,Reports)
reports, Reports = readSheet()

# This 'widget' is used to give us interactive graph capabilities in the matplotlib figures.
%matplotlib widget

# Handle date time conversions between pandas and matplotlib.
from pandas.plotting import register_matplotlib_converters
register_matplotlib_converters()

#
def head(filename,N=10):
    n=0
    with open(Path(filename)) as fp:
        for l in fp.readlines():
            if n<N:
                print(l,end="")
                n=n+1

#
def tail(filename,N=10):
    n=0
    with open(Path(filename)) as fp:
        lines=[i for i in fp.readlines()]
        Nlines=len(lines)
        for l in lines[Nlines-N:]:
            if n<N:
                print(l,end="")
                n=n+1

#
def ConvertNumeric(df,var):
    df[var]= pd.to_numeric(df[var],errors='coerce')

#
def testLogger(filename):
    Sep=","
    Data=[]
    with open(filename) as fp:
        for line in fp.readlines():
            row=line.split(Sep)
            Data.append(len(row))
    print(pd.DataFrame({"Columns":Data}).describe())

#
def ncolumnsLogger(filename):
    Sep=","
    Data=[]
    with open(filename) as fp:
        for line in fp.readlines():
            row=line.split(Sep)
            Data.append(len(row))
    return(np.median(np.array(Data)))

#
def readLogger(filename, nLines=18):
    DataString=""
    Sep=","
    with open(filename) as fp:
        for line in fp.readlines():
            row=line.split(Sep)
            if isinstance(nLines, list):
                if len(row) in nLines and re.match('\W*(\d{4})[/.-](\d{2})[/.-](\d{2})\s{1}(\d{2}):(\d{2}):(\d{2})\W*',row[0]):
                    DataString=DataString+line                
            else:
                if len(row) == nLines and re.match('\W*(\d{4})[/.-](\d{2})[/.-](\d{2})\s{1}(\d{2}):(\d{2}):(\d{2})\W*',row[0]):
                    DataString=DataString+line
    return(DataString)

#
def readRM(filename, nLines=18):
    DataString=""
    Sep=","
    with open(filename) as fp:
        for line in fp.readlines():
            row=line.split(Sep)
            if len(row) == nLines and re.match('\W*(\d{4})[/.-](\d{2})[/.-](\d{2})\s{1}(\d{2}):(\d{2}):(\d{2})\W*',row[0]):
                DataString=DataString+line
    return(DataString)

# List all the files in the specified directory containing the given extention.
def lsR(dir,pattern=".csv",prefix="",postfix=""):
    list=[]
    for path,dirs,files in os.walk(Path(dir)):
        for f in files:
            if f.count(pattern) > 0 and f.startswith(prefix) and f.endswith(postfix):
                list.append(os.path.join(path,f))
    return(list)

#
def clearplots():
    plt.clf()

def instruments(nr):
    for i in df.columns:
        if i.startswith('{}'.format(nr)):
            print("'{}',".format(i))

Directory pathways

Back to the top

# Set the main DropBox directory.
DropboxDir=Path("/home/roelof/Dropbox (NWU)")

# Set the main data directory pathway. This is where the main data files are stored.
DataDir=Path(DropboxDir / "CRG_Projects/01_Research_Campaigns/2021_Low-CostSensorSouthAfrica/01_FieldCampaigns")

# Set a pathway for where the figures/images will be saved.
FigDir=Path(DropboxDir / "CRG_Projects/01_Research_Campaigns/2021_Low-CostSensorSouthAfrica/03_DataAnalysisOutputs/Figures")

# Set a pathway for where the tables/data files will be saved.
TabDir=Path(DropboxDir / "CRG_Projects/01_Research_Campaigns/2021_Low-CostSensorSouthAfrica/03_DataAnalysisOutputs/Tables")

QCQA = Path(DropboxDir/'CRG_Projects/01_Research_Campaigns/2021_Low-CostSensorSouthAfrica/03_DataAnalysisOutputs/QCQA_DATA')

Import data

InstrumentNames=["00_Reference_SAWS_df0",
                 "01_ES642_U16486_df1",
                "02_ES642_U16489_df2",
                "03_ARISense_SN000-57_df3",
                "04_ARISense_SN000-59_df4",
                "05_Vaisala_S1830003_df5",
                "06_S500_5002-2D82-001_df6",
                "07_S500_ECM-1906191-003_df7",
                "08_PolludroneSmart_EA01P0001_df8",
                "09_SimplicityV1_CCSENV011_df9",
                "10_SimplicityV1_CCSENV020_df10",
                "11_SimplicityV2_IMTAQS0001_df11",
                "12_SimplicityV2_IMTAQS0002_df12",
                "13_SimplicityV2_IMTAQS0003_df13",
                "14_SimplicityV2_IMTAQS0004_df14",
                "15_ICOMSMART_20149_df15",
                 "16_RAMP_173_df16",
                "17_GM-5000_CM21035290_df17",
                "18_Plantower_xxx_df18",
                "19_Plantower_xxx_df19",
                "20_Zephyr_642-SA_df20",
                "21_Zephyr_729-SA_df21",
                "22_Atmos_84CCA8B167D2_df22",
                "23_Atmos_98F4ABDCA328_df23",
                #"24_Dylos_xxx_df24",
                #"25_RAMP_xxx_df25",
                '26_Zephyr_533_df26' 
                "27_PolludroneSmart_PM01P0007_df27",
                "28_RAMP_177_df28"]

df=pd.DataFrame()
Files = [f for f in sorted(os.listdir(TabDir)) if f.startswith("df") and 'level1' not in f]

for f in Files:
    print(f)
    dftmp=pd.read_csv(TabDir/f, parse_dates=[0], index_col=0 )
    dftmp=dftmp.resample('10min').mean()
    df = pd.concat([df,dftmp], axis=1)
    
df=df.loc['2021-04-26':'2021-10-29']

df0.csv
df1.csv
df10.csv
df11.csv
df12.csv
df13.csv
df14.csv
df15.csv
df17.csv
df18.csv
df2.csv
df20.csv
df21.csv
df22.csv
df23.csv
df24.csv
df26.csv
df27.csv
df3.csv
df4.csv
df5.csv
df6.csv
df7.csv
df8.csv
df9.csv

Quality Control

Back to the top

1. Vaisala, Aeroqual (PM), Aeroqual (CO) - MRC
2. Polludrone Smart (1), Polludrone Smart (2), GM-5000- eroAfrica
3. Simplicity V1 (1), Simplicity V1 (2), Simplicity V2 (1), Simplicity V2(2), Simplicity V2 (3), Simplicity V2 (4) - Simplicity
4. ECOMSMART - Envirocon
5. Planttower (1),Planttower (2) - Sensors.Africa
6. Zephyr (1), Zephyr (2) - UL
7. Atmos (1), Atmos (2) - SkySide
8. Zypher (3) - Sedulitas
9. RAMP (1), RAMP (2) - AfriqAir

Apply report changes to data

https://docs.google.com/spreadsheets/d/1Fe1P2QEiCjl7KK6YEckzGLC_VqOuGtaChaDGbn2SOhE/edit?usp=sharing

# Update google sheet
reports, Reports = readSheet()

Units={"Ambient_Relative_Humidity":"(%)",
    "Humidity":"(%)",
    "Internal_Battery":"(%)",
    "Internal_Relative_Humidity":"(%)",
    "Relative_Humidity":"(%)",
    "Internal_Temp":"(C)",
    "Noise":"(dB)",
    "Wind_Direction":"(deg)",
    "Ambient_Temperature":"(degC)",
    "Internal_Temperature":"(degC)",
    "Sensor_Temperature":"(degC)",
    "Temperature":"(degC)",
    "Wind_Direction":"(degrees)",
    "Winds_Speed":"(degrees)",
    "Atmospheric_Pressure":"(hPa)",
    "Pressure":"(hPa)",
    "Flow":"(l/m)",
    "CO":"(mg/m3)",
    "Rain":"(mm)",
    "Wind_Direction":"(m/s)",
    "Wind_Speed":"(m/s)",
    "H2S":"(ppb)",
    "NO2":"(ppb)",
    "NO":"(ppb)",
    "NOx":"(ppb)",
    "O3":"(ppb)",
    "SO2":"(ppb)",
    "TVOC":"(ppb)",
    "CO2":"(ppm)",
    "CO":"(ppm)",
    "eCO2":"(ppm)",
    "H2S":"(ug/m3)",
    "NO2":"(ug/m3)",
    "NO":"(ug/m3)",
    "O3":"(ug/m3)",
    "PM10":"(ug/m3)",
    "PM1":"(ug/m3)",
    "PM2.5":"(ug/m3)",
    "SO2":"(ug/m3)",
    "Line_Voltage":"(V)",
    "Solar_Radiation":"(W/m2)"
}

for index,row in Reports.iterrows():
    ins='What is the instrument?'
    var='Which instrument variable failed?'
    sdate='When did the instrument start failing?'
    stime='What time did the instrument start to fail?'
    edate='When was the instrument failure fixed?'
    etime='What time was the instrument failure fixed?'
    
    S=pd.to_datetime("{} {}".format(row[sdate],row[stime])).tz_localize('Africa/Johannesburg')
    E=pd.to_datetime("{} {}".format(row[edate],row[etime])).tz_localize('Africa/Johannesburg')

    v="{:02.0f}_{}".format(int(row[ins].replace("df","")),row[var])
    
    if v in df.columns:
        print("Correcting ",v)
        df.loc[(df.index >= S)&(df.index <= E), v] = nan   
    elif 'All' in v:
        print("Correcting ",v)
        x = [i for i in df.columns if i.startswith('{}'.format(v[:2]))]
        df.loc[(df.index >= S)&(df.index <= E), x] = nan
    else:
        print("{} not in dataset".format(v))

Correcting  00_SO2 (ppb)
Correcting  05_All
Correcting  05_All
Correcting  06_PM2.5 (ug/m3)
Correcting  00_Horiba_SO2 (ppb)
Correcting  05_PM2.5 (ug/m3)
Correcting  05_PM10 (ug/m3)
Correcting  00_PM10 (ug/m3)
Correcting  00_PM10 (ug/m3)
Correcting  00_NO (ppb)
Correcting  00_NO2 (ppb)
Correcting  00_NOx (ppb)
Correcting  00_NO (ppb)
Correcting  00_NO2 (ppb)
Correcting  00_NOx (ppb)
Correcting  03_NO (ppb)
Correcting  00_Horiba_SO2 (ppb)
Correcting  00_SO2 (ppb)
Correcting  00_SO2 (ppb)
Correcting  00_SO2 (ppb)
Correcting  00_SO2 (ppb)
Correcting  00_SO2 (ppb)
Correcting  00_SO2 (ppb)
Correcting  00_SO2 (ppb)

Remove Zero and Span & Site Vists

sitev = ['4/26/2021','5/3/2021','5/6/2021','5/6/2021','5/10/2021',
        '5/17/2021',
        '5/21/2021',
        '5/25/2021',
        '5/30/2021',
        '6/15/2021',
        '6/21/2021',
         '7/2/2021',
        '7/5/2021',
        '7/19/2021',
        '7/23/2021',
        '7/26/2021',
        '8/2/2021',
        '8/2/2021',
        '8/10/2021',
        '8/16/2021',
        '8/16/2021',
        '8/23/2021',
        '8/26/2021',
        '8/31/2021',
         '9/6/2021',
        '9/9/2021',
        '9/13/2021',
        '9/20/2021',
        '9/23/2021',
        '10/29/2021',
        '4/26/2021',
        '5/3/2021',
        '5/6/2021',
        '5/6/2021',
        '5/10/2021',
        '5/17/2021',
        '5/21/2021',
        '5/25/2021',
        '5/30/2021',
        '6/15/2021',
        '6/21/2021',
        '7/2/2021',
        '7/5/2021',
        '7/19/2021',
        '7/23/2021',
        '7/26/2021',
        '8/2/2021',
        '8/2/2021',
        '8/10/2021',
        '8/16/2021',
        '8/16/2021',
        '8/23/2021',
        '8/26/2021',
        '8/31/2021',
        '9/6/2021',
        '9/9/2021',
        '9/13/2021',
        '9/20/2021',
        '9/23/2021',
        '10/5/2021',
        '10/11/2021',
        '10/29/2021']

sitev = pd.to_datetime(sitev, format='%m/%d/%Y')

sitev = sitev.strftime('%Y-%m-%d')
sitev = sitev.tolist()

df[sitev] = nan

/home/roelof/env/conda-python3-env/lib/python3.10/site-packages/pandas/core/frame.py:3645: FutureWarning: Indexing a DataFrame with a datetimelike index using a single string to slice the rows, like `frame[string]`, is deprecated and will be removed in a future version. Use `frame.loc[string]` instead.
  self[col] = value

Reference instruments

reference = ['00_Wind_Speed (m/s)',
'00_Wind_Direction (deg)',
'00_Ambient_Temp (degC)',
'00_Relative_Humidity (%)',
'00_Solar_Radiation (W/m2)',
'00_Atmospheric_Pressure (hPa)',
'00_Internal_Temp (degC)',
'00_Line_Voltage (V)',
'00_Rain (mm)',
'00_SO2 (ppb)',
'00_PM10 (ug/m3)',
'00_PM2.5 (ug/m3)',
'00_NO (ppb)',
'00_NO2 (ppb)',
'00_NOx (ppb)',
'00_O3 (ppb)',
'00_CO (ppm)',
'00_Horiba_SO2 (ppb)']

df[reference].columns

Index(['00_Wind_Speed (m/s)', '00_Wind_Direction (deg)',
       '00_Ambient_Temp (degC)', '00_Relative_Humidity (%)',
       '00_Solar_Radiation (W/m2)', '00_Atmospheric_Pressure (hPa)',
       '00_Internal_Temp (degC)', '00_Line_Voltage (V)', '00_Rain (mm)',
       '00_SO2 (ppb)', '00_PM10 (ug/m3)', '00_PM2.5 (ug/m3)', '00_NO (ppb)',
       '00_NO2 (ppb)', '00_NOx (ppb)', '00_O3 (ppb)', '00_CO (ppm)',
       '00_Horiba_SO2 (ppb)'],
      dtype='object')

PVars = ['00_SO2 (ppb)', '00_PM10 (ug/m3)', '00_PM2.5 (ug/m3)', '00_NO (ppb)',
       '00_NO2 (ppb)', '00_NOx (ppb)', '00_O3 (ppb)', '00_CO (ppm)',
       '00_Horiba_SO2 (ppb)']

%matplotlib inline

Quick Plot

df[PVars].plot(subplots=True, figsize=(15,15))

array([<AxesSubplot:>, <AxesSubplot:>, <AxesSubplot:>, <AxesSubplot:>,
       <AxesSubplot:>, <AxesSubplot:>, <AxesSubplot:>, <AxesSubplot:>,
       <AxesSubplot:>], dtype=object)

Write to QCQA Directory

#df[reference].to_csv(Path(QCQA/'df0.csv'), float_format="%.3f")

#os.listdir(Path(DropboxDir/'CRG_Projects/01_Research_Campaigns/2021_Low-CostSensorSouthAfrica/03_DataAnalysisOutputs/QCQA_DATA'))

#dftest = pd.read_csv(QCQA/'df0.csv', index_col=0, parse_dates=True)

#dftest.describe()

#dftest.plot(subplots=True, figsize=(15,15))

MRC

MRC = ['05_SO2 (ppb)',
'05_CO (ppm)',
'05_O3 (ppb)',
'05_PM2.5 (ug/m3)',
'05_PM10 (ug/m3)',
'05_Ambient_Temperature (degC)',
'05_Relative_Humidity (%)',
'05_Pressure (hPa)',
'05_Validity','06_PM10 (ug/m3)',
'06_PM2.5 (ug/m3)','07_CO (ppm)']

DF05 "05_Vaisala_S1830003_df5"

df.info()

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 26928 entries, 2021-04-26 00:00:00+02:00 to 2021-10-29 23:50:00+02:00
Freq: 10T
Columns: 263 entries, 00_Wind_Speed (m/s) to 09_SO2 (ppb)
dtypes: float64(263)
memory usage: 54.2 MB

TimeSeries Plots

variables = ['05_SO2 (ppb)',
                '05_CO (ppm)',
                '05_O3 (ppb)',
                '05_PM2.5 (ug/m3)',
                '05_PM10 (ug/m3)']

for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','Vaisala_S1830003']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)
    #Plot
#    plt.tight_layout()
#    plt.show()
    
    #Save
    x = re.split('_|\s',i)
#    plt.savefig(Path(QCQA/'TimeSeries_05_Vaisala_S1830003_df5_{}.png'.format(x[1])), dpi=300)

#clearplots()

Scatterplots

variables = ['05_SO2 (ppb)',
                '05_CO (ppm)',
                '05_O3 (ppb)',
                '05_PM2.5 (ug/m3)',
                '05_PM10 (ug/m3)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='Vaisala_S1830003 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='Vaisala_S1830003 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
  

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
#    plt.savefig(Path(QCQA/'ScatterPlot_05_Vaisala_S1830003_df5_{}.png'.format(x[1])), dpi=300)

Accuracy Metrics

variables = ['05_SO2 (ppb)',
                '05_CO (ppm)',
                '05_O3 (ppb)',
                '05_PM2.5 (ug/m3)',
                '05_PM10 (ug/m3)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,4, figsize=(10,10))
    
    #Create Dataframe with hourly and daily statistics 
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    hrmse = mean_squared_error(Xhr, Yhr, squared=False)
    drmse = mean_squared_error(Xday, Yday, squared=False)
    stats = {'Slope':[hslope, dslope],'Intercept':[hintercept,dintercept],'Pearson R':[hr_value,dr_value], 'RMSE':[hrmse, drmse]}
    df_Stats = pd.DataFrame(stats,index=['Hourly','Daily'])
    
    # Add x-axis and y-axis
    axs[0].plot(df_Stats.index, df_Stats['Slope'])
    axs[1].plot(df_Stats.index, df_Stats['Intercept'])
    axs[2].plot(df_Stats.index, df_Stats['Pearson R'])
    axs[3].plot(df_Stats.index, df_Stats['RMSE'])
    
    #Set title and labels for axis
    axs[0].set(xlabel='Average Interval',
         title='Slope')
    axs[1].set(xlabel='Average Interval',
         title='Intercept')
    axs[2].set(xlabel='Average Interval',
         title='Pearson R')
    axs[3].set(xlabel='Average Interval',
         title='RMSE')
    
    fig.suptitle('Vaisala_S1830003 {} - FRM/FEM Accuracy'.format(i[3:]), fontsize=16)

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
 #   x = re.split('_|\s',i)
    
 #   plt.savefig(Path(QCQA/'ScatterPlot_05_Vaisala_S1830003_df5_{}.png'.format(x[1])), dpi=300)

DF06 "06_S500_5002-2D82-001_df6"

TimeSeries Plots

for i in df.columns:
    if i.startswith('06'):
        print(i)

06_PM10 (ug/m3)
06_PM2.5 (ug/m3)

variables = ['06_PM10 (ug/m3)',
            '06_PM2.5 (ug/m3)']

for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','S500_5002-2D82-001']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)

    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
#    plt.savefig(Path(QCQA/'TimeSeries_06_S500_5002-2D82-001_df6_{}.png'.format(x[1])), dpi=300)

#clearplots()

Scatterplots

variables = ['06_PM10 (ug/m3)',
            '06_PM2.5 (ug/m3)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='S500_5002-2D82-001 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='S500_5002-2D82-001 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
  

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
 #   plt.savefig(Path(QCQA/'ScatterPlot_06_S500_5002-2D82-001_df6_{}.png'.format(x[1])), dpi=300)

DF07 "07_S500_ECM-1906191-003_df7"

Timeseries

for i in df.columns:
    if i.startswith('07'):
        print(i)

07_CO (ppm)

variables = ['07_CO (ppm)']

for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','S500_ECM-1906191-003']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)

    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
#    plt.savefig(Path(QCQA/'TimeSeries_07_S500_ECM-1906191-003_df7_{}.png'.format(x[1])), dpi=300)

Scatterplot

variables = ['07_CO (ppm)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='S500_ECM-1906191-003 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='S500_ECM-1906191-003 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
  

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
#    plt.savefig(Path(QCQA/'ScatterPlot_07_S500_ECM-1906191-003_df7_{}.png'.format(x[1])), dpi=300)

QCQA

PosixPath('/home/roelof/Dropbox (NWU)/CRG_Projects/01_Research_Campaigns/2021_Low-CostSensorSouthAfrica/03_DataAnalysisOutputs/QCQA_DATA')

eroAfrica

eroAfrica = ['08_CO2 (ppm)',
'08_CO (ppm)',
'08_NO2 (ppb)',
'08_O3 (ppb)',
'08_NO (ppb)',
'08_SO2 (ppb)',
'08_Relative_Humidity (%)',
'08_PM2.5 (ug/m3)',
'08_PM10 (ug/m3)',
'08_Ambient_Temperature (degC)','27_CO2 (ppm)',
'27_CO (ppm)',
'27_NO2 (ppb)',
'27_O3 (ppb)',
'27_NO (ppb)',
'27_SO2 (ppb)',
'27_Relative_Humidity (%)',
'27_PM2.5 (ug/m3)',
'27_PM10 (ug/m3)',
'27_Ambient_Temperature (degC)','17_CO (ppm)',
'17_O3 (ppb)',
'17_NO2 (ppb)',
'17_SO2 (ppb)',
'17_NO (ppb)',
'17_PM2.5 (ug/m3)',
'17_PM10 (ug/m3)',
'17_Relative Humidity (%)']

DF08 "08_PolludroneSmart_EA01P0001_df8"

Timeseries

variables = ['08_CO (ppm)',
                '08_NO2 (ppb)',
                '08_O3 (ppb)',
                '08_NO (ppb)',
                '08_SO2 (ppb)',
                '08_PM2.5 (ug/m3)',
                '08_PM10 (ug/m3)']

for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','PolludroneSmart_EA01P0001']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)

    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
#    plt.savefig(Path(QCQA/'TimeSeries_08_PolludroneSmart_EA01P0001_df8_{}.png'.format(x[1])), dpi=300)

Scatterplot

variables = ['08_CO (ppm)',
                '08_NO2 (ppb)',
                '08_O3 (ppb)',
                '08_NO (ppb)',
                '08_SO2 (ppb)',
                '08_PM2.5 (ug/m3)',
                '08_PM10 (ug/m3)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='PolludroneSmart_EA01P0001 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='PolludroneSmart_EA01P0001 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
  

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
 #   plt.savefig(Path(QCQA/'ScatterPlot_08_PolludroneSmart_EA01P0001_df8_{}.png'.format(x[1])), dpi=300)

DF17 "17_GM-5000_CM21035290_df17"

Timeseries

variables = ['17_O3 (ppb)',
            '17_NO2 (ppb)',
            '17_SO2 (ppb)',
            '17_NO (ppb)',
            '17_PM2.5 (ug/m3)',
            '17_PM10 (ug/m3)']

for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','GM-5000_CM21035290']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)

    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
#    plt.savefig(Path(QCQA/'TimeSeries_17_GM-5000_CM21035290_df17_{}.png'.format(x[1])), dpi=300)

Scatterplot

variables = ['17_O3 (ppb)',
            '17_NO2 (ppb)',
            '17_SO2 (ppb)',
            '17_NO (ppb)',
            '17_PM2.5 (ug/m3)',
            '17_PM10 (ug/m3)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='GM-5000_CM21035290 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='GM-5000_CM21035290 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
  

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
 #   plt.savefig(Path(QCQA/'ScatterPlot_17_GM-5000_CM21035290_df17_{}.png'.format(x[1])), dpi=300)

DF27 "27_PolludroneSmart_PM01P0007_df27"

Timeseries

variables = ['27_CO (ppm)',
            '27_NO2 (ppb)',
            '27_O3 (ppb)',
            '27_NO (ppb)',
            '27_SO2 (ppb)',
            '27_PM2.5 (ug/m3)',
            '27_PM10 (ug/m3)']

for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','PolludroneSmart_PM01P0007_df27']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)

    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
 #   plt.savefig(Path(QCQA/'TimeSeries_27_PolludroneSmart_PM01P0007_df27_{}.png'.format(x[1])), dpi=300)

Scatterplots

variables = ['27_CO (ppm)',
            '27_NO2 (ppb)',
            '27_O3 (ppb)',
            '27_NO (ppb)',
            '27_SO2 (ppb)',
            '27_PM2.5 (ug/m3)',
            '27_PM10 (ug/m3)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='PolludroneSmart_PM01P0007 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='PolludroneSmart_PM01P0007 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
  

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
 #   plt.savefig(Path(QCQA/'ScatterPlot_27_PolludroneSmart_PM01P0007_df27_{}.png'.format(x[1])), dpi=300)

Simplicity

Simplicity = ['09_CO (ppm)',
'09_Pressure (hPa)',
'09_TVOC (ppb)',
'09_eCO2 (ppm)',
'09_Internal_Temperature (degC)',
'09_Ambient_Temperature (degC)',
'09_Internal_Relative_Humidity (%)',
'09_Ambient_Relative_Humidity (%)',
'09_PM1 (ug/m3)',
'09_PM2.5 (ug/m3)',
'09_PM10 (ug/m3)',
'09_NO2 (ppb)',
'09_O3 (ppb)',
'09_SO2 (ppb)',
'10_CO (ppm)',
'10_Pressure (hPa)',
'10_TVOC (ppb)',
'10_eCO2 (ppm)',
'10_Internal_Temperature (degC)',
'10_Ambient_Temperature (degC)',
'10_Internal_Relative_Humidity (%)',
'10_Ambient_Relative_Humidity (%)',
'10_PM1 (ug/m3)',
'10_PM2.5 (ug/m3)',
'10_PM10 (ug/m3)',
'10_NO2 (ppb)',
'10_O3 (ppb)',
'10_SO2 (ppb)',
'11_CO (ppm)',
'11_Pressure (hPa)',
'11_TVOC (ppb)',
'11_eCO2 (ppm)',
'11_Internal_Temperature (degC)',
'11_Ambient_Temperature (degC)',
'11_Internal_Relative_Humidity (%)',
'11_Ambient_Relative_Humidity (%)',
'11_PM1 (ug/m3)',
'11_PM2.5 (ug/m3)',
'11_PM10 (ug/m3)',
'11_NO2 (ppb)',
'11_O3 (ppb)',
'11_SO2 (ppb)',
'12_CO (ppm)',
'12_Pressure (hPa)',
'12_TVOC (ppb)',
'12_eCO2 (ppm)',
'12_Internal_Temperature (degC)',
'12_Ambient_Temperature (degC)',
'12_Internal_Relative_Humidity (%)',
'12_Ambient_Relative_Humidity (%)',
'12_PM1 (ug/m3)',
'12_PM2.5 (ug/m3)',
'12_PM10 (ug/m3)',
'12_NO2 (ppb)',
'12_O3 (ppb)',
'12_SO2 (ppb)',
'13_CO (ppm)',
'13_Pressure (hPa)',
'13_TVOC (ppb)',
'13_eCO2 (ppm)',
'13_Internal_Temperature (degC)',
'13_Ambient_Temperature (degC)',
'13_Internal_Relative_Humidity (%)',
'13_Ambient_Relative_Humidity (%)',
'13_PM1 (ug/m3)',
'13_PM2.5 (ug/m3)',
'13_PM10 (ug/m3)',
'13_NO2 (ppb)',
'13_O3 (ppb)',
'13_SO2 (ppb)',
'14_CO (ppm)',
'14_Pressure (hPa)',
'14_TVOC (ppb)',
'14_eCO2 (ppm)',
'14_Internal_Temperature (degC)',
'14_Ambient_Temperature (degC)',
'14_Internal_Relative_Humidity (%)',
'14_Ambient_Relative_Humidity (%)',
'14_PM1 (ug/m3)',
'14_PM2.5 (ug/m3)',
'14_PM10 (ug/m3)',
'14_NO2 (ppb)',
'14_O3 (ppb)',
'14_SO2 (ppb)']

"09_SimplicityV1_CCSENV011_df9"

instruments('09')

'09_CO (ppm)',
'09_Pressure (hPa)',
'09_TVOC (ppb)',
'09_eCO2 (ppm)',
'09_Internal_Temperature (degC)',
'09_Ambient_Temperature (degC)',
'09_Internal_Relative_Humidity (%)',
'09_Ambient_Relative_Humidity (%)',
'09_PM1 (ug/m3)',
'09_PM2.5 (ug/m3)',
'09_PM10 (ug/m3)',
'09_NO2 (ppb)',
'09_O3 (ppb)',
'09_SO2 (ppb)',

Timeseries

variables = ['09_PM2.5 (ug/m3)',
                '09_PM10 (ug/m3)',
                '09_NO2 (ppb)',
                '09_O3 (ppb)',
                '09_SO2 (ppb)','09_CO (ppm)' ]

for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','SimplicityV1_CCSENV011']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)

    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
 #   plt.savefig(Path(QCQA/'TimeSeries_09_SimplicityV1_CCSENV011_df9_{}.png'.format(x[1])), dpi=300)

Scatterplots

variables = ['09_PM2.5 (ug/m3)',
                '09_PM10 (ug/m3)',
                '09_NO2 (ppb)',
                '09_O3 (ppb)',
                '09_SO2 (ppb)','09_CO (ppm)' ]

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='SimplicityV1_CCSENV011 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='SimplicityV1_CCSENV011 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
  

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
 #   plt.savefig(Path(QCQA/'ScatterPlot_09_SimplicityV1_CCSENV011_df9_{}.png'.format(x[1])), dpi=300)

"10_SimplicityV1_CCSENV020_df10"

instruments('10')

'10_CO (ppm)',
'10_Pressure (hPa)',
'10_TVOC (ppb)',
'10_eCO2 (ppm)',
'10_Internal_Temperature (degC)',
'10_Ambient_Temperature (degC)',
'10_Internal_Relative_Humidity (%)',
'10_Ambient_Relative_Humidity (%)',
'10_PM1 (ug/m3)',
'10_PM2.5 (ug/m3)',
'10_PM10 (ug/m3)',
'10_NO2 (ppb)',
'10_O3 (ppb)',
'10_SO2 (ppb)',

Timeseries

variables = ['10_PM2.5 (ug/m3)',
            '10_PM10 (ug/m3)',
            '10_NO2 (ppb)',
            '10_O3 (ppb)',
            '10_SO2 (ppb)','10_CO (ppm)']

for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','SimplicityV1_CCSENV020']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)

    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
 #   plt.savefig(Path(QCQA/'TimeSeries_10_SimplicityV1_CCSENV020_df10_{}.png'.format(x[1])), dpi=300)

Scatterplots

variables = ['10_PM2.5 (ug/m3)',
            '10_PM10 (ug/m3)',
            '10_NO2 (ppb)',
            '10_O3 (ppb)',
            '10_SO2 (ppb)','10_CO (ppm)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='SimplicityV1_CCSENV020 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='SimplicityV1_CCSENV020 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
  

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
  #  plt.savefig(Path(QCQA/'ScatterPlot_10_SimplicityV1_CCSENV020_df10_{}.png'.format(x[1])), dpi=300)

"11_SimplicityV2_IMTAQS0001_df11"

instruments('11')

'11_CO (ppm)',
'11_Pressure (hPa)',
'11_TVOC (ppb)',
'11_eCO2 (ppm)',
'11_Internal_Temperature (degC)',
'11_Ambient_Temperature (degC)',
'11_Internal_Relative_Humidity (%)',
'11_Ambient_Relative_Humidity (%)',
'11_PM1 (ug/m3)',
'11_PM2.5 (ug/m3)',
'11_PM10 (ug/m3)',
'11_NO2 (ppb)',
'11_O3 (ppb)',
'11_SO2 (ppb)',

Timeseries

variables = ['11_PM2.5 (ug/m3)',
                '11_PM10 (ug/m3)',
                '11_NO2 (ppb)',
                '11_O3 (ppb)',
                '11_SO2 (ppb)','11_CO (ppm)' ]

for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','SimplicityV2_IMTAQS0001']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)

    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
  #  plt.savefig(Path(QCQA/'TimeSeries_11_SimplicityV2_IMTAQS0001_df11_{}.png'.format(x[1])), dpi=300)

Scatterplots

variables = ['11_PM2.5 (ug/m3)',
                '11_PM10 (ug/m3)',
                '11_NO2 (ppb)',
                '11_O3 (ppb)',
                '11_SO2 (ppb)','11_CO (ppm)'  ]

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='SimplicityV2_IMTAQS0001 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='SimplicityV2_IMTAQS0001 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
  

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
  #  plt.savefig(Path(QCQA/'ScatterPlot_11_SimplicityV2_IMTAQS0001_df11_{}.png'.format(x[1])), dpi=300)

"12_SimplicityV2_IMTAQS0002_df12"

instruments('12')

'12_CO (ppm)',
'12_Pressure (hPa)',
'12_TVOC (ppb)',
'12_eCO2 (ppm)',
'12_Internal_Temperature (degC)',
'12_Ambient_Temperature (degC)',
'12_Internal_Relative_Humidity (%)',
'12_Ambient_Relative_Humidity (%)',
'12_PM1 (ug/m3)',
'12_PM2.5 (ug/m3)',
'12_PM10 (ug/m3)',
'12_NO2 (ppb)',
'12_O3 (ppb)',
'12_SO2 (ppb)',

Timeseries

variables = ['12_PM2.5 (ug/m3)',
                '12_PM10 (ug/m3)',
                '12_NO2 (ppb)',
                '12_O3 (ppb)',
                '12_SO2 (ppb)','12_CO (ppm)' ]

for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','SimplicityV2_IMTAQS0002']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)

    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
 #   plt.savefig(Path(QCQA/'TimeSeries_12_SimplicityV2_IMTAQS0002_df12_{}.png'.format(x[1])), dpi=300)

Scatterplots

variables = ['12_PM2.5 (ug/m3)',
                '12_PM10 (ug/m3)',
                '12_NO2 (ppb)',
                '12_O3 (ppb)',
                '12_SO2 (ppb)','12_CO (ppm)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='SimplicityV2_IMTAQS0002 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='SimplicityV2_IMTAQS0002 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
  

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
  #  plt.savefig(Path(QCQA/'ScatterPlot_12_SimplicityV2_IMTAQS0002_df12_{}.png'.format(x[1])), dpi=300)

"13_SimplicityV2_IMTAQS0003_df13"

instruments('13')

'13_CO (ppm)',
'13_Pressure (hPa)',
'13_TVOC (ppb)',
'13_eCO2 (ppm)',
'13_Internal_Temperature (degC)',
'13_Ambient_Temperature (degC)',
'13_Internal_Relative_Humidity (%)',
'13_Ambient_Relative_Humidity (%)',
'13_PM1 (ug/m3)',
'13_PM2.5 (ug/m3)',
'13_PM10 (ug/m3)',
'13_NO2 (ppb)',
'13_O3 (ppb)',
'13_SO2 (ppb)',

Timeseries

variables = ['13_PM2.5 (ug/m3)',
                '13_PM10 (ug/m3)',
                '13_NO2 (ppb)',
                '13_O3 (ppb)',
                '13_SO2 (ppb)','13_CO (ppm)' ]

for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','SimplicityV2_IMTAQS0003']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)

    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
 #   plt.savefig(Path(QCQA/'TimeSeries_13_SimplicityV2_IMTAQS0003_df13_{}.png'.format(x[1])), dpi=300)

Scatterplots

variables = ['09_PM2.5 (ug/m3)',
                '09_PM10 (ug/m3)',
                '09_NO2 (ppb)',
                '09_O3 (ppb)',
                '09_SO2 (ppb)','09_CO (ppm)' ]

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='SimplicityV2_IMTAQS0003 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='SimplicityV2_IMTAQS0003 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
  

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
 #   plt.savefig(Path(QCQA/'ScatterPlot_13_SimplicityV2_IMTAQS0003_df13_{}.png'.format(x[1])), dpi=300)

"14_SimplicityV2_IMTAQS0004_df14"

instruments('14')

'14_CO (ppm)',
'14_Pressure (hPa)',
'14_TVOC (ppb)',
'14_eCO2 (ppm)',
'14_Internal_Temperature (degC)',
'14_Ambient_Temperature (degC)',
'14_Internal_Relative_Humidity (%)',
'14_Ambient_Relative_Humidity (%)',
'14_PM1 (ug/m3)',
'14_PM2.5 (ug/m3)',
'14_PM10 (ug/m3)',
'14_NO2 (ppb)',
'14_O3 (ppb)',
'14_SO2 (ppb)',

Timeseries

variables = ['14_PM2.5 (ug/m3)',
                '14_PM10 (ug/m3)',
                '14_NO2 (ppb)',
                '14_O3 (ppb)',
                '14_SO2 (ppb)','14_CO (ppm)' ]

for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','SimplicityV2_IMTAQS0004']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)

    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
  #  plt.savefig(Path(QCQA/'TimeSeries_14_SimplicityV2_IMTAQS0004_df14_{}.png'.format(x[1])), dpi=300)

Scatterplots

variables = ['14_PM2.5 (ug/m3)',
                '14_PM10 (ug/m3)',
                '14_NO2 (ppb)',
                '14_O3 (ppb)',
                '14_SO2 (ppb)','14_CO (ppm)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='SimplicityV2_IMTAQS0004 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='SimplicityV2_IMTAQS0004 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
  

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
  #  plt.savefig(Path(QCQA/'ScatterPlot_14_SimplicityV2_IMTAQS0004_df14_{}.png'.format(x[1])), dpi=300)

Envirocon

Envirocon = ['15_Ambient_Temperature (degC)',
'15_Relative_Humidity (%)',
'15_Pressure (hPa)',
'15_SO2 (ppb)',
'15_NO2 (ppb)',
'15_O3 (ppb)',
'15_PM1 (ug/m3)',
'15_PM2.5 (ug/m3)',
'15_PM10 (ug/m3)',
'15_Internal_Battery (%)',
'15_Wind_Direction (m/s)',
'15_Winds_Speed (degrees)']

DF15 "15_ICOMSMART_20149_df15"

Timeseries

variables = ['15_SO2 (ppb)',
'15_NO2 (ppb)',
'15_O3 (ppb)',
'15_PM2.5 (ug/m3)',
'15_PM10 (ug/m3)']

for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','ICOMSMART_20149']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)

    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
 #   plt.savefig(Path(QCQA/'TimeSeries_15_ICOMSMART_20149_df15_{}.png'.format(x[1])), dpi=300)

Scatterplots

variables = ['15_SO2 (ppb)',
'15_NO2 (ppb)',
'15_O3 (ppb)',
'15_PM2.5 (ug/m3)',
'15_PM10 (ug/m3)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='ICOMSMART_20149 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='ICOMSMART_20149 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
  

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
    plt.savefig(Path(QCQA/'ScatterPlot_15_ICOMSMART_20149_df15_{}.png'.format(x[1])), dpi=300)

<Figure size 432x288 with 0 Axes>

<Figure size 432x288 with 0 Axes>

<Figure size 432x288 with 0 Axes>

<Figure size 432x288 with 0 Axes>

<Figure size 432x288 with 0 Axes>

Sensors.Africa

Sensors_Africa = ['18_PM1 (ug/m3)',
'18_PM10 (ug/m3)',
'18_PM2.5 (ug/m3)',
'18_Relative_Humidity (%)',
'18_Ambient_Temperature (degC)']

DF18 "18_Plantower_108_df18"

Timeseries

variables = ['18_PM10 (ug/m3)','18_PM2.5 (ug/m3)']

for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','Plantower_108']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)
    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
 #   plt.savefig(Path(QCQA/'TimeSeries_18_Plantower_108_df18_{}.png'.format(x[1])), dpi=300)

Scatterplots

variables = ['18_PM10 (ug/m3)','18_PM2.5 (ug/m3)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='Plantower_108 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='Plantower_108 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
  

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
 #   plt.savefig(Path(QCQA/'ScatterPlot_18_Plantower_108_df18_{}.png'.format(x[1])), dpi=300)

University of L

UL = ['20_NO2 (ug/m3)',
'20_O3 (ug/m3)',
'20_NO (ug/m3)',
'20_PM2.5 (ug/m3)',
'20_PM1 (ug/m3)',
'20_SO2 (ug/m3)',
'20_PM10 (ug/m3)',
'20_CO (mg/m3)',
'20_H2S (ug/m3)',
'20_Ambient_Temperature (degC)',
'20_Relative_Humidity (%)',
'20_Pressure (hPa)',
'20_NO2 (ppb)',
'20_O3 (ppb)',
'20_CO (ppm)',
'20_NO (ppb)',
'20_H2S (ppb)',
'21_NO2 (ug/m3)',
'21_O3 (ug/m3)',
'21_NO (ug/m3)',
'21_SO2 (ug/m3)',
'21_PM1 (ug/m3)',
'21_PM2.5 (ug/m3)',
'21_PM10 (ug/m3)',
'21_CO (mg/m3)',
'21_H2S (ug/m3)',
'21_Ambient_Temperature (degC)',
'21_Relative_Humidity (%)',
'21_Pressure (hPa)',
'21_NO2 (ppb)',
'21_SO2 (ppb)',
'21_O3 (ppb)',
'21_CO (ppm)',
'21_NO (ppb)',
'21_H2S (ppb)']

DF20 "20_Zephyr_642-SA_df20"

instruments('20')

'20_NO2 (ug/m3)',
'20_O3 (ug/m3)',
'20_NO (ug/m3)',
'20_PM2.5 (ug/m3)',
'20_PM1 (ug/m3)',
'20_SO2 (ug/m3)',
'20_PM10 (ug/m3)',
'20_CO (mg/m3)',
'20_H2S (ug/m3)',
'20_Ambient_Temperature (degC)',
'20_Relative_Humidity (%)',
'20_Pressure (hPa)',
'20_NO2 (ppb)',
'20_O3 (ppb)',
'20_CO (ppm)',
'20_NO (ppb)',
'20_H2S (ppb)',

Timeseries

variables = ['20_NO2 (ppb)',
'20_O3 (ppb)',
'20_CO (ppm)',
'20_NO (ppb)','20_PM2.5 (ug/m3)',
'20_PM10 (ug/m3)']
    
    
for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','Zephyr_642-SA']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)
    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
#    plt.savefig(Path(QCQA/'TimeSeries_20_Zephyr_642-SA_df20_{}.png'.format(x[1])), dpi=300)

Scatterplots

variables = ['20_NO2 (ppb)',
'20_O3 (ppb)',
'20_CO (ppm)',
'20_NO (ppb)','20_PM2.5 (ug/m3)',
'20_PM10 (ug/m3)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='Zephyr_642-SA {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='Zephyr_642-SA {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
    

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
#    plt.savefig(Path(QCQA/'ScatterPlot_20_Zephyr_642-SA_df20_{}.png'.format(x[1])), dpi=300)

DF21 "21_Zephyr_729-SA_df21"

Timeseries

instruments('21')

'21_NO2 (ug/m3)',
'21_O3 (ug/m3)',
'21_NO (ug/m3)',
'21_SO2 (ug/m3)',
'21_PM1 (ug/m3)',
'21_PM2.5 (ug/m3)',
'21_PM10 (ug/m3)',
'21_CO (mg/m3)',
'21_H2S (ug/m3)',
'21_Ambient_Temperature (degC)',
'21_Relative_Humidity (%)',
'21_Pressure (hPa)',
'21_NO2 (ppb)',
'21_SO2 (ppb)',
'21_O3 (ppb)',
'21_CO (ppm)',
'21_NO (ppb)',
'21_H2S (ppb)',

variables = ['21_NO2 (ppb)',
'21_SO2 (ppb)',
'21_O3 (ppb)',
'21_CO (ppm)',
'21_NO (ppb)','21_PM2.5 (ug/m3)',
'21_PM10 (ug/m3)']
    
    
for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','Zephyr_729-SA']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)
    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
 #   plt.savefig(Path(QCQA/'TimeSeries_21_Zephyr_729-SA_df21_{}.png'.format(x[1])), dpi=300)

Scatterplots

variables = ['21_NO2 (ppb)',
'21_SO2 (ppb)',
'21_O3 (ppb)',
'21_CO (ppm)',
'21_NO (ppb)','21_PM2.5 (ug/m3)',
'21_PM10 (ug/m3)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='Zephyr_729-SA {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='Zephyr_729-SA {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
    

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
 #   plt.savefig(Path(QCQA/'ScatterPlot_21_Zephyr_729-SA_df21_{}.png'.format(x[1])), dpi=300)

Skyside

SkySide = ['22_PM1 (ug/m3)',
'22_PM2.5 (ug/m3)',
'22_PM10 (ug/m3)',
'22_Relative_Humidity (%)',
'22_Ambient_Temperature (degC)',
'23_PM1 (ug/m3)',
'23_PM2.5 (ug/m3)',
'23_PM10 (ug/m3)',
'23_Relative_Humidity (%)',
'23_Ambient_Temperature (degC)']

DF22 "22_Atmos_84CCA8B167D2_df22"

variables = ['22_PM2.5 (ug/m3)',
'22_PM10 (ug/m3)']
    
    
for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','Atmos_84CCA8B167D2']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)
    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
 #   plt.savefig(Path(QCQA/'TimeSeries_22_Atmos_84CCA8B167D2_df22_{}.png'.format(x[1])), dpi=300)

Scatterplots

variables = ['22_PM2.5 (ug/m3)',
'22_PM10 (ug/m3)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='Atmos_84CCA8B167D2 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='Atmos_84CCA8B167D2 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
    

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
 #   plt.savefig(Path(QCQA/'ScatterPlot_22_Atmos_84CCA8B167D2_df22_{}.png'.format(x[1])), dpi=300)

DF23 "23_Atmos_98F4ABDCA328_df23"

variables = ['23_PM2.5 (ug/m3)',
'23_PM10 (ug/m3)']
    
    
for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','Atmos_98F4ABDCA328']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)
    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
 #   plt.savefig(Path(QCQA/'TimeSeries_23_Atmos_98F4ABDCA328_df23_{}.png'.format(x[1])), dpi=300)

Scatterplots

variables = ['23_PM2.5 (ug/m3)',
'23_PM10 (ug/m3)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='Atmos_98F4ABDCA328 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='Atmos_98F4ABDCA328 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
    

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
  #  plt.savefig(Path(QCQA/'ScatterPlot_23_Atmos_98F4ABDCA328_df23_{}.png'.format(x[1])), dpi=300)

Sedulitas

instruments('26')

'26_NO2 (ug/m3)',
'26_O3 (ug/m3)',
'26_NO (ug/m3)',
'26_SO2 (ug/m3)',
'26_PM1 (ug/m3)',
'26_PM2.5 (ug/m3)',
'26_PM10 (ug/m3)',
'26_CO (mg/m3)',
'26_H2S (ug/m3)',
'26_Ambient_Temperature (degC)',
'26_Relative_Humidity (%)',
'26_Pressure (hPa)',
'26_NO2 (ppb)',
'26_SO2 (ppb)',
'26_O3 (ppb)',
'26_CO (ppm)',
'26_NO (ppb)',
'26_H2S (ppb)',

Sedulitas = ['26_Longitude',
'26_Internal_Temerature (degC)',
'26_Internal_Relative_Humidity (%)',
'26_NO2 (ug/m3)',
'26_O3 (ug/m3)',
'26_NO (ug/m3)',
'26_SO2 (ug/m3)',
'26_PM1 (ug/m3)',
'26_PM2.5 (ug/m3)',
'26_PM10 (ug/m3)',
'26_CO (mg/m3)',
'26_H2S (ug/m3)',
'26_Ambient_Temperature (degC)',
'26_Relative_Humidity (%)',
'26_Pressure (hPa)']

DF26 26_Zephyr_533_df26

variables = ['26_NO2 (ppb)',
'26_SO2 (ppb)',
'26_O3 (ppb)',
'26_CO (ppm)',
'26_NO (ppb)']
    
    
for i in variables:
    #Create figure and plot space
    fig, axs = plt.subplots(2, figsize=(10,10))

    line_labels = ['FRM/FEM','Zephyr_533']
                       
    # Add x-axis and y-axis
    axs[0].plot(df.resample('1H').mean().index.values,
        df[['00_{}'.format(i[3:])]].resample('1H').mean(),
       df[[i]].resample('1H').mean())

    axs[1].plot(df.resample('1D').mean().index.values,
            df[['00_{}'.format(i[3:])]].resample('1D').mean(),
            df[[i]].resample('1D').mean())

    #Set title and labels for axis
    axs[0].set(xlabel='Date',
          ylabel=i[3:],
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Date',
          ylabel=i[3:],
          title='24-hour averaged {}'.format(i[3:]))

    # Set the legend
    fig.legend(labels=line_labels,
           borderaxespad=0.1,
           bbox_to_anchor=(1, 0.5))
    fig.subplots_adjust(hspace=0.3)
    #Plot
    plt.tight_layout()
    plt.show()
    #Save
    x = re.split('_|\s',i)
#    plt.savefig(Path(QCQA/'TimeSeries_26_Zephyr_533_df26_{}.png'.format(x[1])), dpi=300)

Scatterplots

variables = ['26_NO2 (ppb)',
'26_SO2 (ppb)',
'26_O3 (ppb)',
'26_CO (ppm)',
'26_NO (ppb)']

for i in variables:
    
    hr = df[[i,'00_{}'.format(i[3:])]].resample('1H').mean().dropna(how='any')
    
    day = df[[i,'00_{}'.format(i[3:])]].resample('1D').mean().dropna(how='any')
    
    Xhr = hr[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yhr = hr[i].values.reshape(-1,1)
    
    Xday = day[['00_{}'.format(i[3:])]].values.reshape(-1,1)
    Yday = day[i].values.reshape(-1,1)
    
    #Create figure and plot space
    fig, axs = plt.subplots(1,2, figsize=(10,10), sharex=True, sharey=True)
    
    # Add x-axis and y-axis
    axs[0].scatter(Xhr,Yhr, alpha=0.25)
    hslope, hintercept, hr_value, hp_value, hstd_err = linregress(Xhr[:,0], Yhr[:,0])
    axs[0].plot(Xhr, hintercept+hslope*Xhr, 'r')
    axs[0].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(hintercept,hslope, hr_value), fontsize=10, transform=axs[0].transAxes)

    
    axs[1].scatter(Xday,Yday, alpha=0.35)
    dslope, dintercept, dr_value, dp_value, dstd_err = linregress(Xday[:,0], Yday[:,0])
    axs[1].plot(Xday, dintercept+dslope*Xday, 'r')
    axs[1].text(0.6,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(dintercept,dslope, dr_value), fontsize=10, transform=axs[1].transAxes)
    
    #Set title and labels for axis
    axs[0].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='Zephyr_533 {}'.format(i[3:]),
          title='1-hour averaged {}'.format(i[3:]))

    axs[1].set(xlabel='Reference {}'.format(i[3:]),
          ylabel='Zephyr_533 {}'.format(i[3:]),
          title='24-hour averaged {}'.format(i[3:]))
    

    #Plot
    plt.tight_layout()
    plt.show()
   
    #Save
    x = re.split('_|\s',i)
    
 #   plt.savefig(Path(QCQA/'ScatterPlot_26_Zephyr_533_df26_{}.png'.format(x[1])), dpi=300)

AfriqAir

#AfriqAir = []

Data analysis

Back to the top

1. xxx
2. xxx
3. xxx
4. xxx
5. xxx
6. xxx

df.info()

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 26928 entries, 2021-04-26 00:00:00+00:00 to 2021-10-29 23:50:00+00:00
Freq: 10T
Columns: 281 entries, 18_PM1 (ug/m3) to 12_SO2 (ppb)
dtypes: float64(281)
memory usage: 57.9 MB

ax=df[[i for i in df.columns if 'PM10' in i]].resample('1H').mean().plot()
ax.get_legend().remove()

df[[i for i in df.columns if 'PM10' in i]].resample('1H').mean()

	18_PM10 (ug/m3)	06_PM10 (ug/m3)	17_PM10 (ug/m3)	09_PM10 (ug/m3)	11_PM10 (ug/m3)	04_PM10 (ug/m3)	20_PM10 (ug/m3)	27_PM10 (ug/m3)	03_PM10 (ug/m3)	13_PM10 (ug/m3)	...	10_PM10 (ug/m3)	21_PM10 (ug/m3)	00_PM10 (ug/m3)	00_PM10_Status	14_PM10 (ug/m3)	08_PM10 (ug/m3)	26_PM10 (ug/m3)	15_PM10 (ug/m3)	22_PM10 (ug/m3)	12_PM10 (ug/m3)
2021-04-26 00:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-04-26 01:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-04-26 02:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-04-26 03:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-04-26 04:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
2021-10-29 19:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-10-29 20:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-10-29 21:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-10-29 22:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-10-29 23:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN

4488 rows × 22 columns

pm10l=[i for i in df.columns if 'PM10' in i and i != '00_PM10 (ug/m3)']

df[pm10l].resample('1H').mean()

	18_PM10 (ug/m3)	06_PM10 (ug/m3)	17_PM10 (ug/m3)	09_PM10 (ug/m3)	11_PM10 (ug/m3)	04_PM10 (ug/m3)	20_PM10 (ug/m3)	27_PM10 (ug/m3)	03_PM10 (ug/m3)	13_PM10 (ug/m3)	05_PM10 (ug/m3)	23_PM10 (ug/m3)	10_PM10 (ug/m3)	21_PM10 (ug/m3)	14_PM10 (ug/m3)	08_PM10 (ug/m3)	26_PM10 (ug/m3)	15_PM10 (ug/m3)	22_PM10 (ug/m3)	12_PM10 (ug/m3)
2021-04-26 00:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-04-26 01:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-04-26 02:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-04-26 03:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-04-26 04:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
2021-10-29 17:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-10-29 18:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-10-29 19:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-10-29 20:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2021-10-29 21:00:00+00:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN

4486 rows × 20 columns

len([i for i in df.columns if 'PM10' in i and i != '00_PM10 (ug/m3)'])

20

Y=df['00_PM10 (ug/m3)'].resample('1H').mean().values

fig = plt.figure()
ax = fig.add_axes([0,0,1,1])

for p in [i for i in df.columns if 'PM10' in i and i != '00_PM10 (ug/m3)']:
    ax.scatter(df[p].resample('1H').mean().values,Y)
    print(p)

ax.set_ylabel('Reference PM10 (ug/m3)')
ax.set_xlabel('Low cost PM10 (ug/m3)')
ax.set_xlim([0,200])
ax.set_ylim([0,200])

18_PM10 (ug/m3)
06_PM10 (ug/m3)
17_PM10 (ug/m3)
09_PM10 (ug/m3)
11_PM10 (ug/m3)
04_PM10 (ug/m3)
20_PM10 (ug/m3)
27_PM10 (ug/m3)
03_PM10 (ug/m3)
13_PM10 (ug/m3)
05_PM10 (ug/m3)
23_PM10 (ug/m3)
10_PM10 (ug/m3)
21_PM10 (ug/m3)
14_PM10 (ug/m3)
08_PM10 (ug/m3)
26_PM10 (ug/m3)
15_PM10 (ug/m3)
22_PM10 (ug/m3)
12_PM10 (ug/m3)

(0.0, 200.0)

Y=df['00_PM10 (ug/m3)'].resample('1H').mean().values

lcvars=[i for i in df.columns if 'PM10' in i and i != '00_PM10 (ug/m3)']

fig, axs = plt.subplots(5,4, sharex='col',sharey='row',gridspec_kw={'hspace':0.1, 'wspace':0.1}, figsize=(12,12))

for i in range(len(lcvars)):
    dftmp=df[['00_PM10 (ug/m3)',lcvars[i]]].resample('1H').mean()
    dftmp=dftmp.dropna(how='any')
    X=dftmp[lcvars[i]].values
    Y=dftmp['00_PM10 (ug/m3)'].values
    if len(X)>0 and len(Y)>0:
        axs[int(i/4),i%4].scatter(X,Y, alpha=0.25)
        axs[int(i/4),i%4].set_xlim(0,200)
        axs[int(i/4),i%4].set_ylim(0,200)
        res = linregress(X,Y)

        axs[int(i/4),i%4].text(0.3,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(res.intercept,res.slope, res.rvalue), fontsize=10, transform=axs[int(i/4),i%4].transAxes)

        axs[int(i/4),i%4].set_title(lcvars[i].split("_")[0], y=1.0, pad=-14, fontsize=14)
        #axs[int(i/4),i%4].set_ylabel('Reference PM10 (ug/m3)')
        #axs[int(i/4),i%4].set_xlabel('Low cost PM10 (ug/m3)')

        axs[int(i/4),i%4].plot(X, res.intercept+res.slope*X, 'r')
        #print(b,m,r,p)
    
fig.add_subplot(111, frameon=False)
plt.tick_params(labelcolor='none', top=False, bottom=False, left=False, right=False )
plt.grid(False)
plt.xlabel("Low cost PM10 (ug/m3)", fontsize=16)
plt.ylabel("Reference PM10 (ug/m3)", fontsize=16)

#plt.savefig(Path(FigDir / '20210929_CorrMatrix_{}.png'.format('PM10')), dpi=90, bbox_inches='tight', pad_inches=0)

Text(0, 0.5, 'Reference PM10 (ug/m3)')

res

LinregressResult(slope=1.1893391151949624, intercept=22.598520129501793, rvalue=0.5779128274834254, pvalue=0.0, stderr=0.02773893085524807, intercept_stderr=1.0795965150520952)

lcvars

['18_PM10 (ug/m3)',
 '06_PM10 (ug/m3)',
 '17_PM10 (ug/m3)',
 '09_PM10 (ug/m3)',
 '11_PM10 (ug/m3)',
 '04_PM10 (ug/m3)',
 '20_PM10 (ug/m3)',
 '27_PM10 (ug/m3)',
 '03_PM10 (ug/m3)',
 '13_PM10 (ug/m3)',
 '05_PM10 (ug/m3)',
 '23_PM10 (ug/m3)',
 '10_PM10 (ug/m3)',
 '21_PM10 (ug/m3)',
 '14_PM10 (ug/m3)',
 '08_PM10 (ug/m3)',
 '26_PM10 (ug/m3)',
 '15_PM10 (ug/m3)',
 '22_PM10 (ug/m3)',
 '12_PM10 (ug/m3)']

lcvars=[i for i in df.columns if 'PM10' in i]

for i in range(len(lcvars)):
    print(lcvars[i])

18_PM10 (ug/m3)
06_PM10 (ug/m3)
17_PM10 (ug/m3)
09_PM10 (ug/m3)
11_PM10 (ug/m3)
04_PM10 (ug/m3)
20_PM10 (ug/m3)
27_PM10 (ug/m3)
03_PM10 (ug/m3)
13_PM10 (ug/m3)
05_PM10 (ug/m3)
23_PM10 (ug/m3)
10_PM10 (ug/m3)
21_PM10 (ug/m3)
00_PM10 (ug/m3)
14_PM10 (ug/m3)
08_PM10 (ug/m3)
26_PM10 (ug/m3)
15_PM10 (ug/m3)
22_PM10 (ug/m3)
12_PM10 (ug/m3)

%matplotlib inline

for i in df.columns:
    if i.startswith("00"):
        print(i)

00_Wind_Speed (m/s)
00_Wind_Direction (deg)
00_Ambient_Temp (degC)
00_Relative_Humidity (%)
00_Solar_Radiation (W/m2)
00_Atmospheric_Pressure (hPa)
00_Internal_Temp (degC)
00_Line_Voltage (V)
00_Rain (mm)
00_SO2 (ppb)
00_PM10 (ug/m3)
00_PM2.5 (ug/m3)
00_NO (ppb)
00_NO2 (ppb)
00_NOx (ppb)
00_O3 (ppb)
00_CO (ppm)
00_Horiba_SO2 (ppb)

df = df.reindex(sorted(df.columns), axis=1)

Vars=['PM10 (ug/m3)', 'PM2.5 (ug/m3)', 'O3 (ppb)', 'SO2 (ppb)', 'NO2 (ppb)', 'CO (ppm)']

for var in Vars:
    refvar='00_{}'.format(var)
    Y=df[refvar].resample('1H').mean().values

    lcvars=[i for i in df.columns if var in i and i != refvar]

    fig, axs = plt.subplots(6,4,sharex='col',sharey='row',gridspec_kw={'hspace':0.1, 'wspace':0.1}, figsize=(12,12))

    for i in range(len(lcvars)):
        dftmp=df[[refvar,lcvars[i]]].resample('1H').mean()
        dftmp=dftmp.dropna(how='any')
        X=dftmp[lcvars[i]].values
        Y=dftmp[refvar].values
        if len(X)>0 and len(Y)>0:
            axs[int(i/4),i%4].scatter(X,Y, alpha=0.25)
            axs[int(i/4),i%4].set_xlim(0,dftmp[refvar].max())
            axs[int(i/4),i%4].set_ylim(0,dftmp[refvar].max())
            res = linregress(X,Y)

            axs[int(i/4),i%4].text(0.3,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(res.intercept,res.slope, res.rvalue), fontsize=10, transform=axs[int(i/4),i%4].transAxes)

            axs[int(i/4),i%4].set_title(lcvars[i].split("_")[0], y=1.0, pad=-14, fontsize=14)
            #axs[int(i/4),i%4].set_ylabel('Reference PM10 (ug/m3)')
            #axs[int(i/4),i%4].set_xlabel('Low cost PM10 (ug/m3)')

            axs[int(i/4),i%4].plot(X, res.intercept+res.slope*X, 'r')

    fig.add_subplot(111, frameon=False)
    plt.tick_params(labelcolor='none', top=False, bottom=False, left=False, right=False )
    plt.grid(False)
    plt.xlabel("Low cost {}".format(var), fontsize=16)
    plt.ylabel("Reference {}".format(var), fontsize=16)

    #plt.savefig(Path(FigDir / '20210928_AllVars_CorrMatrix_{}.png'.format(var.split()[0])), dpi=90, bbox_inches='tight', pad_inches=0)

Pvars = ["03_PM10 (ug/m3)",
        "04_PM10 (ug/m3)",
        "05_PM10 (ug/m3)",
        "06_PM10 (ug/m3)",
        "08_PM10 (ug/m3)",
        "09_PM10 (ug/m3)",
        "10_PM10 (ug/m3)",
        "11_PM10 (ug/m3)",
        "12_PM10 (ug/m3)",
        "13_PM10 (ug/m3)",
        "14_PM10 (ug/m3)",
        "15_PM10 (ug/m3)",
        "17_PM10 (ug/m3)",
        "18_PM10 (ug/m3)",
        "20_PM10 (ug/m3)",
        "21_PM10 (ug/m3)",
        "22_PM10 (ug/m3)",
        "23_PM10 (ug/m3)",
        "26_PM10 (ug/m3)",
        "27_PM10 (ug/m3)"]

varname="00_PM10 (ug/m3)"
dftmp=df[[varname]].resample('1H').mean()
dftmp['Time'] = dftmp.index.map(lambda x: x.strftime("%H:00"))
diurnal = dftmp[['Time',varname]].groupby('Time').describe()[varname]

fig, ax = plt.subplots(1, figsize=(12,6))
ax.plot(diurnal.index, diurnal['50%'], linewidth=2.0,label=varname, alpha=0.5) 

for var in Pvars:
    dftmp=df[[var]].resample('1H').mean()
    dftmp['Time'] = dftmp.index.map(lambda x: x.strftime("%H:00"))
    diurnal = dftmp[['Time',var]].groupby('Time').describe()[var]
    ax.plot(diurnal.index, diurnal['50%'], linewidth=2.0,label=var, alpha=0.5)
ax.set_xlim(0, 23)                                                                                                                                                                                                                                                          
ax.set_xticks([i for i in range(0,24,3)])                                                                                                                                                                                                                                    

plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))

<matplotlib.legend.Legend at 0x7f7197f52770>

pm10=[i for i in df.columns if 'PM10' in i and i != '00_PM10 (ug/m3)']

%matplotlib widget

#df[pm10].plot(subplots=True, figsize=(20,20))

Diurnal plots

Vars=['PM10 (ug/m3)', 'PM2.5 (ug/m3)', 'O3 (ppb)', 'SO2 (ppb)', 'NO2 (ppb)', 'CO (ppm)']

for var in Vars:
    refvar='00_{}'.format(var)
    dftmp=df[[refvar]].resample('5min').mean()
    dftmp['Time'] = dftmp.index.map(lambda x: x.strftime("%H:00"))
    diurnalref = dftmp[['Time',refvar]].groupby('Time').describe()[refvar]

    lcvars=[i for i in df.columns if var in i and i != refvar]

    fig, axs = plt.subplots(6,4, sharex='col',sharey='row',gridspec_kw={'hspace':0.1, 'wspace':0.1}, figsize=(12,12))

    for i in range(len(lcvars)):
        dftmp=df[[lcvars[i]]].resample('5min').mean()
        dftmp['Time'] = dftmp.index.map(lambda x: x.strftime("%H:00"))
        diurnal = dftmp[['Time',lcvars[i]]].groupby('Time').describe()[lcvars[i]]
        axs[int(i/4),i%4].plot(diurnal.index, diurnal['50%'], linewidth=2.0,label=var, alpha=0.5)
        axs[int(i/4),i%4].plot(diurnalref.index, diurnalref['50%'], linewidth=2.0,label='Reference', alpha=0.5)

        #axs[int(i/4),i%4].set_xlim(0,dftmp[refvar].max())
        #axs[int(i/4),i%4].set_ylim(0,dftmp[refvar].max())
        
        #axs[int(i/4),i%4].text(0.3,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(res.intercept,res.slope, res.rvalue), fontsize=10, transform=axs[int(i/4),i%4].transAxes)

        axs[int(i/4),i%4].set_title(lcvars[i].split("_")[0], y=1.0, pad=-14, fontsize=14)

    for i in range(4):
        axs[4,i%4].set_xlim(0, 23)                                                                                                                                                                                                                                                          
        axs[4,i%4].set_xticks([i for i in range(0,24,6)])

            
    fig.add_subplot(111, frameon=False)
    plt.tick_params(labelcolor='none', top=False, bottom=False, left=False, right=False )
    plt.grid(False)
    plt.xlabel("Time of day", fontsize=16)
    plt.ylabel("Reference (orange) vs low-cost (blue) {}".format(var), fontsize=16)

    #plt.savefig(Path(FigDir / '20210928_Diurnal_{}.png'.format(var.split()[0])), dpi=90, bbox_inches='tight', pad_inches=0)

Data recovery

df.columns

Index(['00_Ambient_Temp (degC)', '00_Atmospheric_Pressure (hPa)',
       '00_CO (ppm)', '00_Horiba_SO2 (ppb)', '00_Internal_Temp (degC)',
       '00_Line_Voltage (V)', '00_NO (ppb)', '00_NO2 (ppb)', '00_NOx (ppb)',
       '00_O3 (ppb)',
       ...
       '27_NO (ppb)', '27_NO2 (ppb)', '27_O3 (ppb)', '27_PM10 (ug/m3)',
       '27_PM2.5 (ug/m3)', '27_Relative_Humidity (%)', '27_SO2 (ppb)',
       'PM_large', 'PM_small', 'Timestamp(UTS)'],
      dtype='object', length=260)

Vars=['PM10 (ug/m3)', 'PM2.5 (ug/m3)', 'O3 (ppb)', 'SO2 (ppb)', 'NO2 (ppb)', 'CO (ppm)']

Instruments=['00',
             '01',
             '02',
             '03',
             '04',
             '05',
             '06',
             '07',
             '08',
             '09',
             '10',
             '11',
             '12',
             '13',
             '14',
             '15',
             '17',
             '20',
             '21',
             '22',
             '23',
             '24',
             '25',
             '26',
             '27']

Table={}

for i in Instruments:
    inst=[]
    for var in Vars:
        varname='{}_{}'.format(i,var)
        if varname in df.columns:
            inst.append("{:.1f}".format(len(df[varname].dropna())/len(df[varname])*100))
            #print(varname,"{:.1f}".format(len(df[varname].dropna())/len(df[varname])*100))
        else:
            inst.append("")
    Table[i]=inst

Table

{'00': ['82.0', '82.0', '51.3', '79.6', '82.0', '82.0'],
 '01': ['', '44.3', '', '', '', ''],
 '02': ['', '79.5', '', '', '', ''],
 '03': ['76.9', '76.9', '76.9', '', '76.9', '12.3'],
 '04': ['44.3', '44.3', '44.3', '', '44.3', '9.3'],
 '05': ['48.7', '49.0', '48.5', '49.1', '', '49.1'],
 '06': ['37.4', '37.4', '', '', '', ''],
 '07': ['', '', '', '', '', '27.1'],
 '08': ['83.2', '82.2', '77.6', '70.5', '81.3', '83.0'],
 '09': ['73.7', '73.7', '73.7', '73.7', '73.7', '73.7'],
 '10': ['69.3', '69.3', '69.3', '69.3', '69.3', '69.3'],
 '11': ['81.9', '81.9', '81.9', '81.9', '81.9', '81.9'],
 '12': ['84.1', '84.1', '84.1', '84.1', '84.1', '84.1'],
 '13': ['84.2', '84.2', '84.2', '84.2', '84.2', '84.2'],
 '14': ['84.1', '84.1', '84.1', '84.1', '84.1', '84.1'],
 '15': ['77.6', '77.6', '77.6', '77.6', '77.6', ''],
 '17': ['75.7', '75.7', '75.7', '75.7', '75.7', '75.7'],
 '20': ['82.1', '82.1', '76.4', '', '82.0', '82.1'],
 '21': ['81.3', '81.3', '80.8', '80.8', '79.7', '81.3'],
 '22': ['61.3', '61.3', '', '', '', ''],
 '23': ['69.0', '69.0', '', '', '', ''],
 '24': ['', '', '', '', '', ''],
 '25': ['', '', '', '', '', ''],
 '26': ['43.3', '43.3', '', '', '', ''],
 '27': ['36.3', '35.9', '28.5', '36.1', '36.3', '36.1']}

dfDA=pd.DataFrame(data=Table).transpose()
dfDA.columns=Vars
dfDA

	PM10 (ug/m3)	PM2.5 (ug/m3)	O3 (ppb)	SO2 (ppb)	NO2 (ppb)	CO (ppm)
00	82.0	82.0	51.3	79.6	82.0	82.0
01		44.3
02		79.5
03	76.9	76.9	76.9		76.9	12.3
04	44.3	44.3	44.3		44.3	9.3
05	48.7	49.0	48.5	49.1		49.1
06	37.4	37.4
07						27.1
08	83.2	82.2	77.6	70.5	81.3	83.0
09	73.7	73.7	73.7	73.7	73.7	73.7
10	69.3	69.3	69.3	69.3	69.3	69.3
11	81.9	81.9	81.9	81.9	81.9	81.9
12	84.1	84.1	84.1	84.1	84.1	84.1
13	84.2	84.2	84.2	84.2	84.2	84.2
14	84.1	84.1	84.1	84.1	84.1	84.1
15	77.6	77.6	77.6	77.6	77.6
17	75.7	75.7	75.7	75.7	75.7	75.7
20	82.1	82.1	76.4		82.0	82.1
21	81.3	81.3	80.8	80.8	79.7	81.3
22	61.3	61.3
23	69.0	69.0
24
25
26	43.3	43.3
27	36.3	35.9	28.5	36.1	36.3	36.1

#dfDA.to_csv(Path(TabDir / '20210829_DataAvailability.csv'),float_format='%.1f')

Factors

Vars=['PM10 (ug/m3)', 'PM2.5 (ug/m3)', 'O3 (ppb)', 'SO2 (ppb)', 'NO2 (ppb)', 'CO (ppm)']

Instruments=['01',
             '02',
             '03',
             '04',
             '05',
             '06',
             '07',
             '08',
             '09',
             '10',
             '11',
             '12',
             '13',
             '14',
             '15',
             '17',
             '18',
             '20',
             '21',
             '22',
             '23',
             '26',
             '27']

Table={}

for i in Instruments:
    inst=[]
    for var in Vars:
        refvar='00_{}'.format(var)
        varname='{}_{}'.format(i,var)
        if varname in df.columns:            
            dftmp=df[[refvar,varname]].resample('1H').mean()
            dftmp=dftmp.dropna(how='any')
            if len(dftmp) > 10:
                X=dftmp[varname].values
                Y=dftmp[refvar].values
                res = linregress(X,Y)
                inst.append("{:.1f}, {:.1f}, {:.2f}".format(res.intercept,res.slope, res.rvalue))
            else:
                inst.append("")
        else:
            inst.append("")
    Table[i]=inst

dfDA=pd.DataFrame(data=Table).transpose()
dfDA.columns=Vars
dfDA

	PM10 (ug/m3)	PM2.5 (ug/m3)	O3 (ppb)	SO2 (ppb)	NO2 (ppb)	CO (ppm)
01		10.4, 0.7, 0.86
02		16.0, 0.5, 0.78
03	49.5, 0.7, 0.28	13.8, 1.5, 0.77	30.7, 0.0, 0.17		11.9, 0.0, 0.03	0.6, 987.2, 0.64
04	56.7, 0.1, 0.08	26.3, 0.5, 0.44	8.2, 0.8, 0.82		11.7, 0.2, 0.25	0.7, 1114.1, 0.72
05	29.0, 0.8, 0.64	28.3, 1.6, 0.27	20.6, 0.3, 0.28	-1.0, 0.1, 0.40		0.5, 1.2, 0.86
06	29.9, 0.0, 0.16	24.6, 0.0, 0.07
07						0.7, 0.2, 0.22
08	51.7, 0.1, 0.22	20.7, 0.4, 0.63	16.9, 0.7, 0.43	-0.4, 1.3, 0.39	7.2, 0.2, 0.26	0.1, 1.8, 0.80
09	24.5, 1.2, 0.56	3.1, 1.2, 0.91	11.8, 0.6, 0.52	2.8, 0.0, 0.21	18.9, -0.0, -0.39	2.4, -0.4, -0.46
10	25.4, 0.8, 0.56	6.0, 0.8, 0.88	15.8, 0.9, 0.48	1.7, 0.0, 0.23	22.6, -0.0, -0.32	-1.2, 0.3, 0.33
11	22.1, 1.1, 0.58	2.9, 1.0, 0.92	14.4, 0.1, 0.45	4.1, 0.0, 0.07	18.4, -0.0, -0.34	1.0, -0.1, -0.31
12	22.6, 1.2, 0.58	3.0, 1.1, 0.91	13.9, 0.1, 0.52	4.6, 0.0, 0.04	19.2, -0.0, -0.38	0.9, -0.6, -0.03
13	23.4, 1.0, 0.57	3.7, 0.9, 0.93	-2.0, 0.4, 0.44	4.0, 0.0, 0.06	21.8, -0.0, -0.38	0.8, 0.0, 0.01
14	22.3, 1.6, 0.60	3.5, 1.4, 0.92	-25.8, 1.1, 0.49	3.4, 0.0, 0.12	19.6, -0.0, -0.37	1.1, -0.1, -0.19
15	22.5, 0.6, 0.65	7.3, 0.5, 0.90	18.4, 0.9, 0.76	5.4, 0.3, 0.16	9.3, 0.1, 0.16
17	27.3, 1.5, 0.60	6.3, 5.7, 0.89	38.0, 0.8, 0.87	7.2, 0.0, 0.09	9.8, 0.1, 0.23	1.3, 1.1, 0.79
18	30.6, 0.7, 0.62	6.9, 0.6, 0.89
20	1.7, 1.6, 0.56	1.0, 0.9, 0.67	10.7, 0.9, 0.92		3.7, 0.6, 0.53	0.3, 0.7, 0.87
21	4.8, 1.2, 0.60	-3.2, 0.8, 0.83	7.9, 0.9, 0.93	-0.6, 0.4, 0.13	6.0, 0.6, 0.50	0.4, 0.7, 0.86
22	25.4, 0.9, 0.56	3.2, 0.9, 0.93
23	33.0, 1.2, 0.54	9.2, 1.0, 0.92
26	19.5, 1.3, 0.46	-4.6, 1.3, 0.86
27	25.5, 1.0, 0.50	12.1, 0.8, 0.77	30.7, 0.1, 0.07	4.4, 0.4, 0.16	-8.0, 0.6, 0.40	-0.1, 1.2, 0.81

#dfDA.to_csv(Path(TabDir / '20210928_CorrelationWithReference.csv'))

Vars=['PM10 (ug/m3)', 'PM2.5 (ug/m3)', 'O3 (ppb)', 'SO2 (ppb)', 'NO2 (ppb)', 'CO (ppm)']

Instruments=['00',
             '01',
             '02',
             '03',
             '04',
             '05',
             '06',
             '07',
             '08',
             '09',
             '10',
             '11',
             '12',
             '13',
             '14',
             '15',
             '17',
             '20',
             '21',
             '22',
             '23',
             '24',
             '25',
             '26',
             '27']

Table={}

for i in Instruments:
    inst=[]
    for var in Vars:
        varname='{}_{}'.format(i,var)
        if varname in df.columns:
            inst.append("{:.1f}, {:.1f}, {:.1f}".format(df[varname].mean(),df[varname].std(),df[varname].quantile(0.99)))
            #print(varname,"{:.1f}".format(len(df[varname].dropna())/len(df[varname])*100))
        else:
            inst.append("")
    Table[i]=inst
    
dfDA=pd.DataFrame(data=Table).transpose()
dfDA.columns=Vars
#dfDA.to_csv(Path(TabDir / '20210829_DataStatistics.csv'))

Vars=['PM10 (ug/m3)', 'PM2.5 (ug/m3)', 'O3 (ppb)', 'SO2 (ppb)', 'NO2 (ppb)', 'CO (ppm)']

Instruments=['00',
             '01',
             '02',
             '03',
             '04',
             '05',
             '06',
             '07',
             '08',
             '09',
             '10',
             '11',
             '12',
             '13',
             '14',
             '15',
             '17',
             '20',
             '21',
             '22',
             '23',
             '24',
             '25',
             '26',
             '27']

Table={}

for i in Instruments:
    inst=[]
    for var in Vars:
        varname='{}_{}'.format(i,var)
        if varname in df.columns:
            inst.append("{:.1f} ({:.1f})".format(df[varname].mean(),df[varname].std()))
            #print(varname,"{:.1f}".format(len(df[varname].dropna())/len(df[varname])*100))
        else:
            inst.append("")
    Table[i]=inst
    
dfDA=pd.DataFrame(data=Table).transpose()
dfDA.columns=Vars
#dfDA.to_csv(Path(TabDir / '20210928_DataMeanSTD.csv'))

dfDA

	PM10 (ug/m3)	PM2.5 (ug/m3)	O3 (ppb)	SO2 (ppb)	NO2 (ppb)	CO (ppm)
00	57.9 (51.7)	30.6 (25.1)	25.3 (17.9)	5.3 (12.0)	11.9 (9.3)	0.9 (0.8)
01		28.9 (31.8)
02		31.8 (41.1)
03	13.8 (20.1)	11.4 (12.8)	-849.1 (472.1)		4.5 (6.0)	0.0 (0.0)
04	23.5 (54.1)	15.8 (24.9)	19.0 (16.7)		15.3 (12.3)	0.0 (0.0)
05	46.7 (44.0)	4.2 (4.4)	23.9 (18.4)	58.3 (44.2)		0.4 (0.6)
06	2567.9 (929.7)	1390.5 (523.8)
07						1.2 (0.7)
08	56.4 (102.8)	26.2 (40.6)	21.5 (15.8)	5.3 (3.7)	28.1 (14.4)	0.4 (0.4)
09	28.3 (24.2)	24.6 (20.2)	26.6 (17.0)	123.0 (137.0)	604.3 (343.3)	3.5 (0.9)
10	34.0 (31.2)	29.3 (27.0)	8.3 (9.9)	138.0 (122.8)	334.8 (97.6)	6.5 (0.7)
11	31.6 (26.9)	27.3 (22.8)	92.7 (65.6)	1136.1 (834.3)	877.4 (432.2)	1.4 (2.4)
12	29.5 (25.0)	25.4 (21.1)	95.2 (80.5)	1292.8 (891.0)	891.8 (430.3)	0.0 (0.0)
13	35.0 (29.9)	29.2 (25.3)	77.7 (23.4)	934.9 (568.2)	481.7 (174.7)	2.4 (0.9)
14	22.1 (19.4)	18.6 (15.8)	47.1 (9.2)	3769.7 (2916.1)	946.7 (426.7)	3.0 (2.3)
15	57.2 (52.5)	44.6 (42.3)	15.2 (26.3)	0.8 (6.1)	27.2 (16.6)
17	21.9 (20.6)	4.5 (4.0)	-13.4 (20.8)	-55.8 (32.5)	20.8 (16.2)	-0.4 (0.6)
20	33.7 (17.7)	31.0 (17.6)	20.6 (19.6)		12.9 (7.9)	0.7 (0.9)
21	44.7 (26.5)	43.5 (26.7)	23.7 (20.7)	13.0 (3.3)	9.6 (7.4)	0.6 (1.0)
22	35.1 (32.4)	29.0 (25.0)
23	22.2 (23.4)	22.1 (23.3)
24
25
26	33.0 (20.6)	24.4 (13.3)
27	31.7 (28.9)	17.4 (17.4)	34.5 (27.3)	7.7 (4.4)	25.7 (4.7)	0.4 (0.3)

Reference air quality

for i in df.columns:
    if i.startswith('00'):
        print(i)

00_Ambient_Temp (degC)
00_Atmospheric_Pressure (hPa)
00_CO (ppm)
00_Horiba_SO2 (ppb)
00_Internal_Temp (degC)
00_Line_Voltage (V)
00_NO (ppb)
00_NO2 (ppb)
00_NOx (ppb)
00_O3 (ppb)
00_PM10 (ug/m3)
00_PM2.5 (ug/m3)
00_Rain (mm)
00_Relative_Humidity (%)
00_SO2 (ppb)
00_Solar_Radiation (W/m2)
00_Wind_Direction (deg)
00_Wind_Speed (m/s)

df[['00_Relative_Humidity (%)','00_Ambient_Temp (degC)']].plot(subplots=True,figsize=(12,12))
#plt.savefig(Path(FigDir / 'Ref_T_and_RH.png'), dpi=90, bbox_inches='tight', pad_inches=0)

array([<AxesSubplot:>, <AxesSubplot:>], dtype=object)

for i in df.columns:
    if i.startswith('00'):
        print(i)

00_Ambient_Temp (degC)
00_Atmospheric_Pressure (hPa)
00_CO (ppm)
00_Horiba_SO2 (ppb)
00_Internal_Temp (degC)
00_Line_Voltage (V)
00_NO (ppb)
00_NO2 (ppb)
00_NOx (ppb)
00_O3 (ppb)
00_PM10 (ug/m3)
00_PM2.5 (ug/m3)
00_Rain (mm)
00_Relative_Humidity (%)
00_SO2 (ppb)
00_Solar_Radiation (W/m2)
00_Wind_Direction (deg)
00_Wind_Speed (m/s)

df[['00_SO2 (ppb)']].plot(subplots=True,figsize=(12,12))
#plt.savefig(Path(FigDir / '20210928_Ref_SO2.png'), dpi=90, bbox_inches='tight', pad_inches=0)

array([<AxesSubplot:>], dtype=object)

df[['00_PM10 (ug/m3)','00_PM2.5 (ug/m3)','00_SO2 (ppb)', '00_CO (ppm)', '00_O3 (ppb)', '00_NO2 (ppb)']].plot(subplots=True,figsize=(12,12))
#plt.savefig(Path(FigDir / '20210928_Ref_aqs.png'), dpi=90, bbox_inches='tight', pad_inches=0)

array([<AxesSubplot:>, <AxesSubplot:>, <AxesSubplot:>, <AxesSubplot:>,
       <AxesSubplot:>, <AxesSubplot:>], dtype=object)

VarNames=['00_PM10 (ug/m3)', '00_PM2.5 (ug/m3)', '00_O3 (ppb)', '00_SO2 (ppb)', '00_NO2 (ppb)', '00_CO (ppm)']
VarNames.append('Time')

VarNames+['Time']

['00_PM10 (ug/m3)',
 '00_PM2.5 (ug/m3)',
 '00_O3 (ppb)',
 '00_SO2 (ppb)',
 '00_NO2 (ppb)',
 '00_CO (ppm)',
 'Time',
 'Time']

VarNames=['00_PM10 (ug/m3)', '00_PM2.5 (ug/m3)', '00_O3 (ppb)', '00_SO2 (ppb)', '00_NO2 (ppb)', '00_CO (ppm)']

dftmp=df[VarNames].resample('5min').mean()
dftmp['Time'] = dftmp.index.map(lambda x: x.strftime("%H:00"))
diurnal = dftmp[['Time']+VarNames].groupby('Time').describe()[VarNames]

diurnal['00_PM10 (ug/m3)']['50%']

Time
00:00    49.87250
01:00    48.53625
02:00    48.59250
03:00    43.88750
04:00    42.18500
05:00    43.95750
06:00    43.70250
07:00    44.38000
08:00    55.16500
09:00    59.36625
10:00    46.50250
11:00    36.95500
12:00    30.54375
13:00    29.61750
14:00    29.15750
15:00    28.45125
16:00    30.31250
17:00    37.58875
18:00    48.25750
19:00    50.94875
20:00    54.12625
21:00    52.67625
22:00    52.39000
23:00    49.81000
Name: 50%, dtype: float64

VarNames=['00_PM10 (ug/m3)', '00_PM2.5 (ug/m3)', '00_O3 (ppb)', '00_SO2 (ppb)', '00_NO2 (ppb)', '00_CO (ppm)']

for row in range(len(VarNames)):
    print(row,diurnal[VarNames[i]]['50%'].values)

0 [49.8725  48.53625 48.5925  43.8875  42.185   43.9575  43.7025  44.38
 55.165   59.36625 46.5025  36.955   30.54375 29.6175  29.1575  28.45125
 30.3125  37.58875 48.2575  50.94875 54.12625 52.67625 52.39    49.81   ]
1 [49.8725  48.53625 48.5925  43.8875  42.185   43.9575  43.7025  44.38
 55.165   59.36625 46.5025  36.955   30.54375 29.6175  29.1575  28.45125
 30.3125  37.58875 48.2575  50.94875 54.12625 52.67625 52.39    49.81   ]
2 [49.8725  48.53625 48.5925  43.8875  42.185   43.9575  43.7025  44.38
 55.165   59.36625 46.5025  36.955   30.54375 29.6175  29.1575  28.45125
 30.3125  37.58875 48.2575  50.94875 54.12625 52.67625 52.39    49.81   ]
3 [49.8725  48.53625 48.5925  43.8875  42.185   43.9575  43.7025  44.38
 55.165   59.36625 46.5025  36.955   30.54375 29.6175  29.1575  28.45125
 30.3125  37.58875 48.2575  50.94875 54.12625 52.67625 52.39    49.81   ]
4 [49.8725  48.53625 48.5925  43.8875  42.185   43.9575  43.7025  44.38
 55.165   59.36625 46.5025  36.955   30.54375 29.6175  29.1575  28.45125
 30.3125  37.58875 48.2575  50.94875 54.12625 52.67625 52.39    49.81   ]
5 [49.8725  48.53625 48.5925  43.8875  42.185   43.9575  43.7025  44.38
 55.165   59.36625 46.5025  36.955   30.54375 29.6175  29.1575  28.45125
 30.3125  37.58875 48.2575  50.94875 54.12625 52.67625 52.39    49.81   ]

fig, axs = plt.subplots(6,1, sharex='col',sharey='row',gridspec_kw={'hspace':0.1, 'wspace':0.1}, figsize=(12,12))

VarNames=['00_PM10 (ug/m3)', '00_PM2.5 (ug/m3)', '00_O3 (ppb)', '00_SO2 (ppb)', '00_NO2 (ppb)', '00_CO (ppm)']

dftmp=df[VarNames].resample('5min').mean()
dftmp['Time'] = dftmp.index.map(lambda x: x.strftime("%H:00"))
diurnal = dftmp[['Time']+VarNames].groupby('Time').describe()[VarNames]

for i in range(len(VarNames)):
    axs[i].plot(diurnal.index, diurnal[VarNames[i]]['50%'].values, linewidth=2.0,label=VarNames[i], alpha=0.5, color='b')
    #axs[i,0].plot(diurnalref.index, diurnal['50%'], linewidth=2.0,label='Reference', alpha=0.5)
    axs[i].plot(diurnal.index, diurnal[VarNames[i]]['75%'], color='b')
    axs[i].plot(diurnal.index, diurnal[VarNames[i]]['25%'], color='b')
    axs[i].fill_between(diurnal.index, diurnal[VarNames[i]]['50%'], diurnal[VarNames[i]]['75%'], alpha=.25, facecolor='b')
    axs[i].fill_between(diurnal.index, diurnal[VarNames[i]]['50%'], diurnal[VarNames[i]]['25%'], alpha=.25, facecolor='b')


    axs[i].set_ylabel(VarNames[i].replace("00_",""))
#axs[int(i/4),i%4].set_xlim(0,dftmp[refvar].max())
#axs[int(i/4),i%4].set_ylim(0,dftmp[refvar].max())

#axs[int(i/4),i%4].text(0.3,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(res.intercept,res.slope, res.rvalue), fontsize=10, transform=axs[int(i/4),i%4].transAxes)

#axs[int(i/4),i%4].set_title(lcvars[i].split("_")[0], y=1.0, pad=-14, fontsize=14)

axs[i].set_xlim(0, 23)                                                                                                                                                                                                                                                          
axs[i].set_xticks([i for i in range(0,24,3)])


fig.add_subplot(111, frameon=False)
plt.tick_params(labelcolor='none', top=False, bottom=False, left=False, right=False )
plt.grid(False)
plt.xlabel("Time of day", fontsize=16)
#plt.ylabel("Reference (orange) vs low-cost (blue) {}".format(VarNames[i]), fontsize=16)

#plt.savefig(Path(FigDir / '20210928_Diurnal_refvars.png'), dpi=90, bbox_inches='tight', pad_inches=0)

Text(0.5, 0, 'Time of day')

'00_Relative_Humidity (%)','00_Ambient_Temp (degC)'

('00_Relative_Humidity (%)', '00_Ambient_Temp (degC)')

fig, axs = plt.subplots(2,1, sharex='col',sharey='row',gridspec_kw={'hspace':0.1, 'wspace':0.1}, figsize=(12,6))

VarNames=['00_Relative_Humidity (%)','00_Ambient_Temp (degC)']

dftmp=df[VarNames].resample('5min').mean()
dftmp['Time'] = dftmp.index.map(lambda x: x.strftime("%H:00"))
diurnal = dftmp[['Time']+VarNames].groupby('Time').describe()[VarNames]

for i in range(len(VarNames)):
    axs[i].plot(diurnal.index, diurnal[VarNames[i]]['50%'].values, linewidth=2.0,label=VarNames[i], alpha=0.5, color='b')
    #axs[i,0].plot(diurnalref.index, diurnal['50%'], linewidth=2.0,label='Reference', alpha=0.5)
    axs[i].plot(diurnal.index, diurnal[VarNames[i]]['75%'], color='b')
    axs[i].plot(diurnal.index, diurnal[VarNames[i]]['25%'], color='b')
    axs[i].fill_between(diurnal.index, diurnal[VarNames[i]]['50%'], diurnal[VarNames[i]]['75%'], alpha=.25, facecolor='b')
    axs[i].fill_between(diurnal.index, diurnal[VarNames[i]]['50%'], diurnal[VarNames[i]]['25%'], alpha=.25, facecolor='b')


    axs[i].set_ylabel(VarNames[i].replace("00_",""))
#axs[int(i/4),i%4].set_xlim(0,dftmp[refvar].max())
#axs[int(i/4),i%4].set_ylim(0,dftmp[refvar].max())

#axs[int(i/4),i%4].text(0.3,0.1,"y={:.1f}x+{:.1f} ({:.2f})".format(res.intercept,res.slope, res.rvalue), fontsize=10, transform=axs[int(i/4),i%4].transAxes)

#axs[int(i/4),i%4].set_title(lcvars[i].split("_")[0], y=1.0, pad=-14, fontsize=14)

axs[i].set_xlim(0, 23)                                                                                                                                                                                                                                                          
axs[i].set_xticks([i for i in range(0,24,3)])


fig.add_subplot(111, frameon=False)
plt.tick_params(labelcolor='none', top=False, bottom=False, left=False, right=False )
plt.grid(False)
plt.xlabel("Time of day", fontsize=16)
#plt.ylabel("Reference (orange) vs low-cost (blue) {}".format(VarNames[i]), fontsize=16)

#plt.savefig(Path(FigDir / '20210928_Diurnal_refTandRH.png'), dpi=90, bbox_inches='tight', pad_inches=0)

Text(0.5, 0, 'Time of day')

Bivariate plots

# Define the settings for the windrose that is being created.
def plot_ppollrose(dfc,WD='00_Wind_Direction (deg)',WS='00_Wind_Speed (m/s)',CO='00_PM10 (ug/m3)',title=""):                                                                                                                                                                                                                                                   
    dfc=dfc[[WD,WS,CO]].dropna()
    wd_rad = np.radians(np.array(dfc[WD]))
    Conc = np.array(dfc[CO], dtype=float)
    WSmax = max(dfc[WS])
    WD1, WS1 = np.meshgrid(np.linspace(0, 2*np.pi, 360), np.linspace(min(dfc[WS]), max(dfc[WS]), 16))
    Z = interpolate.griddata((wd_rad, dfc[WS]), dfc[CO], (WD1, WS1), method='linear')

    levels = MaxNLocator(nbins=10).tick_values(0, 500)
    
    cmap = plt.get_cmap('viridis')
    norm = BoundaryNorm(levels, ncolors=cmap.N, clip=True)
    
    fig, ax = plt.subplots(figsize=(4,4), subplot_kw={"projection": "polar"})
    
    img = ax.pcolormesh(WD1, WS1, Z, cmap=cmap, norm=norm, edgecolor='none', linewidth=0, alpha=1, shading='nearest', snap='bool', capstyle='round')
    
    cbar = plt.colorbar(img, cax=None, ax=None, use_gridspec=True, orientation='vertical', drawedges=True, pad=0.09, shrink=0.5)
    cbar.ax.tick_params(labelsize=10)
    cbar.set_label(CO,size=10, labelpad=+10 )
    
    #To display the grid in the background.
    ax.grid(True)
    # Specify different settings for major and minor grids
    #ax.grid(which = 'minor', alpha = 0.3)
    #ax.grid(which = 'major', alpha = 0.7)
    
    major_yticks = np.arange(0, 15, step=2.5)
    #minor_yticks = np.arange(0, 15, step=0.5)
    #major_xticks = np.arange(0, 8, step=360)
    #minor_xticks = np.arange(0, 360, step=22.5)
    
    ax.tick_params(axis = 'both', which = 'major', labelsize = 10, width=1.5, direction='out', pad=0.8, labelrotation=0, top=True, grid_color='white', grid_linewidth=1)
    ax.tick_params(axis = 'both', which = 'minor', labelsize = 0)
    
    ax.set_yticks(major_yticks)
    ax.set_ylim([0,WSmax])
    #ax.set_yticks(minor_yticks, minor=True)
    #ax.set_xticks(major_xticks)
    #ax.set_xticks(minor_xticks, minor=True)
   
    # To rename the x-axis from Degrees to Directions.
    ax.set_xticklabels(['N', 'N-E', 'E', 'S-E', 'S', 'S-W', 'W', 'N-W'], fontsize=10)
       
    # To move the 0 Deg from the E-point to the N-point.
    ax.set_theta_zero_location("N")
    ax.set_theta_direction(-1)
    
    ax.set_axisbelow(False)
    fig.suptitle(title)
    plt.tight_layout()
    #plt.savefig(output,dpi=300, bbox_inches = "tight")  
    plt.show()
    #plt.close()

#plot_ppollrose(df.loc['2021-09-15'],CO='00_PM10 (ug/m3)', title='Vaal')